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1.0 Abstract 


A flight test was performed to compare the use of three advanced primary flight and navigation 
display concepts to a baseline, round-dial concept to assess the potential for advanced operations. The 
displays were evaluated during visual and instrument approach procedures including an advanced 
instrument approach resembling a visual airport traffic pattern. Nineteen pilots from three pilot groups, 
reflecting the diverse piloting skills of the General Aviation pilot population, served as evaluation 
subjects. The experiment had two thrusts:. 1) an examination of the capabilities of low-time (i.e., <400 
hours), non-instrument-rated pilots to perform nominal instrument approaches, and 2) an exploration of 
potential advanced Visual Meteorological Conditions (VMC)-like approaches in Instrument 
Meteorological Conditions (IMC). Within this context, advanced display concepts are considered to 
include integrated navigation and primary flight displays with either aircraft attitude flight directors or 
Highway In The Sky (HITS) guidance with and without a synthetic depiction of the external visuals (i.e., 
synthetic vision). Relative to the first thrust, the results indicate that using an advanced display concept, 
as tested herein, low-time, non- instrument-rated pilots can exhibit flight-technical performance, 
subjective workload and situation awareness ratings as good as or better than high-time Instrument Flight 
Rules (IFR)-rated pilots using Baseline Round Dials for a nominal IMC approach. For the second thrust, 
the results indicate advanced VMC-like approaches are feasible in IMC, for all pilot groups tested for 
only the Synthetic Vision System (SVS) advanced display concept. These findings are valid for the 
conditions tested and that this test only examined nominal operations and did not include traffic 
separation, hazardous weather, equipment failures, etc. Other related factors not addressed were pilots’ 
decision-making ability or possible overconfidence. 


2.0 Introduction 

Synthetic Vision Systems (SVS) primary flight displays present a continuous computer-generated 
image of the external scene topography from the perspective of the flight deck, albeit possibly minified, 
creating an electronic means of continuous visual-like information for the pilot/flight crew. When 
combined with integrated guidance symbology, SVS have been shown to significantly increase situation 
awareness (SA) and pilot performance while decreasing workload during operations in Instrument 
Meteorological Conditions (IMC) (Ref. 1-12) compared to traditional flight instrument display concepts. 
Throughout this paper, for brevity, this combination of flight path guidance with SVS terrain information 
will be referred to as SVS displays or the SVS display concept. 

A critical component of SVS displays is the appropriate presentation of terrain to the pilot. 
References 1-4, 6, 9, and 12 examined the effects of two primary elements of terrain portrayal on the 
primary flight display (PFD): 1) variations of digital elevation model (DEM) resolution; and, 2) terrain 
texturing, along with other aspects of SVS displays. Higher resolution DEMs can enable more terrain 
features to be represented in the SVS terrain especially in mountainous areas. Different texturing 
concepts can be employed to convey various types of information to the pilot. One type of texturing, 
referred to as photo-realistic, applies ortho-rectified imagery to produce SVS terrain that appears very 
similar to the outside world, thereby conveying cultural feature information. Other terrain texturing 
concepts employ elevation-based terrain texturing concepts which can enhance the terrain height cues. 
References 1-4, 6, 9, and 12 reported on several generic as well as photo-realistic concepts. References 4 
and 12 expanded the analysis to include interactions between different symbology, both 2-dimensional (2- 
D) and 3-dimensional (3-D), along with various SVS terrain portrayal concepts. Results from these 
studies indicated that significant improvements in S A can be realized over a wide range of DEMs and 
texturing concepts, and established recommended best practices for SVS terrain portrayal. Results from 
References 3, 4, 8, and 12 also indicated that 3-D guidance symbology, that combined a minimal 
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highway-in-the-sky (HITS) depiction along with a flight path vector with guidance, provided significant 
benefits compared to other guidance concepts. 

Traditional IMC operations feature approaches with long straight legs, gradual descents and large 
airspace requirements to accommodate historical surveillance, navigation, and flight technical 
performance considerations. These factors can be particularly restrictive at airports in mountainous areas. 
Several references (see Ref. 3-5, 7, 8, 10, 11, and 12) explored the benefits of SVS displays to enable 
advanced IMC approaches. These advanced IMC approaches resembled Visual Meteorological 
Conditions (VMC)-style approaches requiring more aggressive maneuvering than traditional IMC 
operations. If proven, these VMC-style approaches would allow a drastic reduction in the Instrument 
Flight Rules (IFR) ceiling and visibility landing minima below what would normally be in effect at 
terrain-challenged airports during periods of poor visibility. While it is possible that approach operations 
could be conducted with advanced autopilots, it is considered that keeping the pilots in the aircraft control 
loop provides a superior alternative both for safety of flight as well as potential implementation into the 
GA fleet. Results from these studies indicated that VMC-style approaches may be possible with SVS 
displays. 

The current effort obtained flight test data for advanced displays and approaches not traditionally 
available for General Aviation (GA) operations. This new data evaluated the potential of SVS displays to 
effectively transform IMC flight and enable advanced operations. Unlike previous research in this area 
that endeavored to show a significant improvement due to SVS display technology compared to a 
conventional baseline for specific IMC test conditions, this flight test investigated several points to more 
effectively measure, assess, and inteipret the results. While the “gold standard” of IMC operations was 
included (the Instrument Landing System (ILS) approach), a nominal Visual Flight Rules (VFR) 
approach was also included. This establishes the relationship between IMC and VMC operations within 
several common Human Factors (HF) measures and a metric to evaluate advanced approaches and display 
concepts. Two other conditions were included within the data set of significant note: 1) VFR-rated pilots 
flying an ILS approach in IMC with the baseline round dial display concept, and 2) an VMC-like 
approach flown with the advanced display concepts in simulated IMC. The value of #1 was to establish a 
condition within the data set of unquestionably high risk that unfortunately arises in real-world 
operations. This condition also enables an evaluation of the effect of IFR pilot training across a wide 
range of display concepts. The VMC-like approach (#2) in simulated IMC was included to create a direct 
comparison of out-the-window visibility along with enabling the evaluation of a potentially advanced 
IMC approach with substantial operational benefits. The selection of pilots provided evaluated the effect 
of advanced display concepts on low-time VFR pilots along with low-time and high-time IFR-rated 
pilots. 

The recent advent of affordable electronic displays, with dimensions and other characteristics 
suitable for installation in small airplanes, offers GA avionics and airplane manufacturers more flexibility 
in presentation and placement of cockpit instruments and system controls than ever before possible 
according to the General Aviation Manufacturers Association (GAMA) (Ref. 13). These new 
technologies, which include SVS displays, can provide dramatic improvements in the capabilities of GA 
aiip lanes flown by a single pilot in IMC according to Reference 13. The Federal Aviation Administration 
(FAA) has also published an Advisory Circular (AC) [14] relating to the standardization of advanced 
displays, acknowledging Reference 13, and an AC on the acceptable means of compliance for synthetic 
vision and pathway depictions (Ref. 15). Overall, while AC23-26 (Ref. 15) defines criterion of how SVS 
displays can replace traditional displays, it does so within the realm of improved safety and does not 
discuss potential operational benefits due to this technology. 

This paper documents the results of a flight test that was conducted to examine the hypothesis that 
SVS displays may significantly transform approach operations in IMC flight. The flight test was 
conducted using a representative GA aircraft modified to enable evaluation of four primary display 
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concepts in rapid succession. Those primary display concepts were: 1) a baseline round-dial display 
concept; 2) a single-cue flight director display concept (without SVS terrain); 3) a pathway-based 
guidance display concept (without SVS terrain), and 4) the SVS display concept. The subject pilot 
population was as follows: 1) low-time VFR-rated pilots; 2) pilots just exceeding the minimum training 
and experience requirements to possess an IFR rating; and 3) very high-time, highly experienced IFR- 
rated pilots. The evaluation tasks were: 1) a VFR approach; 2) a nominal Instrument Landing System 
(ILS) approach performed in simulated IMC and 3) a VMC-like approach performed in simulated IMC. 
The research location was one with substantial terrain-induced operational restrictions, especially in IMC. 

Three sub-hypotheses were examined to characterize and quantify the transformation of approach 
operations in IMC flight provided by advanced display concepts, especially those employing SVS. 

1) Using SVS displays, the workload and SA of all pilots flying approaches in IMC will be as 
good as, or better than, the same pilots flying in VMC using baseline round-dial displays. 

It is well-known that the reduced visibility in IMC has a substantial impact on pilot workload and SA 
compared to VMC flight. By demonstrating that this hypothesis is true, one could imply that SVS 
displays significantly transform approach operations in IMC flight by reducing pilot workload and 
increasing SA to levels better than or comparable to VMC. 

2) Using SVS displays, low-time, non-instrument-rated pilots flying nominal ILS approaches in 
IMC will achieve flight technical error, workload, and SA results as good as, or better than, 
ratings from instrument-rated pilots conducting the same ILS approach using baseline, 
round-dial displays. 

It should be noted that this experiment only examined nominal approaches and did not include traffic 
separation, hazardous weather, equipment failures, etc. Also, other important factors, such as objective 
measures of pilots’ decision-making ability or possible overconfidence were not evaluated. However, 
considering the substantial training and high standards to obtain an IFR type rating, demonstrating that 
this hypothesis is true could imply that SVS displays have the potential to significantly transform flying 
for low-time VFR pilots, perhaps even enabling them to fly safely in IMC. 

3) Using SVS displays, all pilots, including low-time, non-instrument-rated pilots, will be able to 
conduct IMC approaches with geometries similar to a VFR pattern with flight technical 
error, workload, and SA results as good as, or better than, results from instrument rated 
pilots conducting ILS approaches using baseline round-dial displays. 

The third hypothesis concerns flight performance using SVS displays to conduct IMC landing 
approaches with geometries other than traditional long, straight-in final approach segments. Again, 
considering the substantial training and high standards required to obtain an IFR type rating, 
demonstrating that this hypothesis is true could imply that SVS displays have the potential to significantly 
transform flying for all pilots, perhaps even enabling low-time VFR pilots to fly approaches with curved 
segments safely in IMC. 


3.0 Method 

A coordinated simulation and flight-test effort was conducted using the NASA Langley Research 
Center Integration Flight Deck simulator (IFD) and the NASA Langley Cessna 206 (C206) aircraft. This 
report presents results from the flight test effort (see Reference 1 1 for the IFD simulator results). 

3.1 Flight Test Aircraft and Apparatus 

A C206 aircraft (see Figure 1) was modified for this flight test through installation of a 10.4 inch 
diagonal Liquid Crystal Display (LCD) research display on the right side of the instrument panel; a 6.4 
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inch LCD research display in the top of the center radio stack; two electro-mechanical, simulator-grade, 
round-dial gauges; two general-purpose computers; and an air data and attitude and heading reference 
system. The two electro-mechanical round-dial gauges were required since the 10.4 inch LCD, that was 
the largest that could be installed, was not large enough to simultaneously display all of the round-dial 
gauges. These modifications established an Evaluation Pilot (EP) station on the right side of the aircraft, 
as shown in Figure 2, as well as the required infrastructure to support research operations. In the 
conventional display configuration, seven 3 inch-diameter Baseline Round Dials (BRDs) displayed 
airspeed, attitude, altitude, turn and bank, heading, vertical speed, and localizer/glideslope course 
deviation indicators (CDIs). To allow rapid comparison with the other displays, most of the round-dial 
gauges were computer-generated and displayed on the 10.4 inch LCD. For the advanced display 
configurations, the research display system enabled evaluations of dual 6.4 inch advanced displays (using 
only a portion of the 10.4 inch LCD), such as a Navigation Display (ND) and Primary Flight Display 
(PFD) with HITS symbology. The conventional display and advanced display concept configurations 
could be changed rapidly in flight through computer configuration files. As configured for this flight test, 
the EP occupied the front right seat of the aircraft, and the safety pilot (SP) occupied the front left seat. 

A Test Engineer (TE) occupied the middle row right seat and was provided with display, keyboard, and 
mouse interfaces to operate the research system and direct the test. The third seating row of the C206 
aircraft was removed. 



Figure 1: NASA Langley Cessna 206. 


3.2 Display Concepts 

Four display concepts were evaluated: 

1) Conventional Baseline Round Dials (hereinafter referred to as BRD); 

2) ND and PFD with Single-Cue Flight Director (hereinafter referred to as SCFD); 

3) ND and PFD with HITS providing Pathway-Based Guidance (hereinafter referred to as PBG); 

4) ND and SYS PFD with HITS (hereinafter referred to as SYS). 
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Photographs of the four display concepts as implemented in the C206 are provided in Figures 2 
through 6. Note that the ND display was invariant for the SCFD, PBG, and SVS advanced display 
concepts. 



Figure 2: Baseline Round-Dial Display Concept 

The SCFD display concept was included in this experiment since it is included in many 
commercially available glass-cockpit displays and offers a significant improvement over the minimum 
required baseline round dials. It also serves as a secondary baseline for comparison with the PBG and 
SVS display concepts. The attitude-based SCFD is driven by a compensatory flight director control law 
as described in Appendix A. 
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Figure 3: Single-Cue Flight Director Display Concept 


Based upon results of research conducted on SVS displays to date, such as References 1 through 12, 
perspective pathway-based guidance symbologies have been used with SVS display concepts to enhance 
integration with the 3-D SVS terrain. This integration not only improves terrain awareness relative to the 
desired route, but also enhances pilots’ overall SA since route and outside terrain are provided in the same 
frame of reference. As a result, the PBG display concept was included to create a clear evaluation of the 
differences between a single-cue attitude -based flight director and pathway-based pursuit-based guidance 
symbology as well as the direct effect of SVS terrain (i.e., the only difference between the PBG and SVS 
display concepts was the SVS terrain). The HITS guidance used for this test is known as the Crow’s-Foot 
tunnel with tadpole guidance cue (driven using a modified pursuit control law) and is described in 
References 3, 4, 8, 9, 11, and 12. Right-angle magenta triads formed the corner points of the desired 
flight path tunnel. The tunnel was set to 2 dots of localizer error and 1 dot of glideslope error and limited 
to 600 ft wide and 350 ft tall. See Appendix B for a description of the HITS guidance system employed 
herein. 
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Figure 4: Pathway-Based Guidance Display Concept 



Figure 5: Synthetic Vision System Display Concept 


Some minor variations from what could be considered to be standard baseline round-dial displays 
were used, including: 1) use of a sky pointer (vs. turn pointer) for the bank-angle indication on the attitude 
indicator; 2) expanded pitch scale on the attitude indicator, resulting in a potentially more precise pitch 
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scale than is common; and, 3) use of graphics displays to present airspeed, attitude, turn and bank, and 
directional gyro gauges with simulator-grade electro-mechanical gauges for altitude and vertical speed. 
When considering the level and quality of pilot training involved in this flight test (discussed below), the 
BRD variations were considered minimal, and that the BRD display concept evaluations were 
representative of pilot performance with conventional displays. 
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Figure 6: Close-up of Display Concepts 


The PFD and ND display concepts were presented over a display area of approximately 6.4 inch 
diagonal. This size display is representative of the smaller range of displays being developed for GA. All 
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of the advanced display concepts incoiporated the same ND, and each PFD used a 60-degree horizontal 
field of regard and a corresponding vertical field of regard of 45 degrees. The track-up ND used the same 
terrain database as the SVS PFD and included: 1) a god’s-eye view of terrain; 2) terrain awareness and 
warning system (TAWS) “peaks mode” speckling on the moving map (see References 16 and 17 for 
background on TAWS); 3) flight path and waypoints; ownship symbol and turn predictor “noodle;” 4) 
route information; 5) traffic symbology; and 6) PFD viewing wedge. Traffic symbology was not driven 
by actual data but used pre-recorded simulated aircraft trajectories that would initiate movement when the 
C206 crossed over a specific start point for the approach. 

The SCFD concept had integrated airspeed and altitude tapes along with a sky-pointer/roll scale and 
a single-cue flight director. The PBG concept included the crow’s foot tunnel pathway or HITS with 
tadpole guidance command and a pitch-quickened velocity vector, as well as air traffic symbology. The 
SVS concept included the features of the PBG concept with a terrain background in place of the blue- 
sky/brown- ground presentation. This particular combination of SVS terrain and guidance symbology was 
selected based on results of previous experiments (Ref. 2-5). Close-up photographs of the four display 
concepts are provided in Figure 6. In order to provide the details of the SVS PFD and ND computer 
generated screen captures are presented in Figures 7a and 7b. 

For all display concepts, maximum displayed localizer and glideslope error was set to 2.0 dots 
deflection for the ILS approaches. Indicated angular error was based on the lateral and vertical distance 
from the centerline of the desired flight path and the distance along-path to a notional ILS navigation 
beam source (i.e., localizer and glideslope) and only displayed for the ILS approach evaluations. 



Figure 7a: Expanded view of SVS Display Concept PFD 
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Figure 7b: Expanded View of Navigation Display 

3.3 SVS Terrain Portrayal 

The SVS terrain portrayal used for this study is referred to as the Elevation-Based Generic (EBG) 
terrain portrayal concept, developed and evaluated in earlier studies (Ref. 1-4, 6, 9, 11, and 12). The 
Digital Elevation Model (DEM) used three arc-second resolution (approximately 295 ft (90m) post- 
spacing) terrain data. Based on results from References 2 and 3, this level of DEM resolution provides 
good terrain awareness during IMC flight. 

The EBG texturing concept consisted of twelve equal-height coloration bands. These bands 
corresponded to different absolute terrain elevation levels. The coloration bands were based on VFR 
sectional charts and applied to the various local-area altitude elevation bands, where lower terrain levels 
were rendered with darker colors, while higher terrain levels were assigned lighter colors. A specific 
shade of green was set to the field elevation of 1,175 ft above Mean Sea Level (MSL). The lightest VFR 
sectional color was set to the highest terrain within 50 nm of the research airport, approximately 4000 ft 
MSL. Cultural features, such as roads and rivers, were included as objects in the terrain database. 
Simulated daylight shading was also used to accentuate the terrain features. An example of the EBG 
texturing concept is provided in Figure 7a. This view was taken from the south east looking back towards 
the airport where the flight test was conducted. 

The airport model included runways with markings along with visually prominent airport buildings. 
All models were overlaid on top of the underlying elevation database. Obstacles greater than 200 ft high 
within 20 nm of the airport were represented by narrow rectangular barber-pole striped objects portraying 
their height and location, as provided within the FAA’s published obstacle database. 

3.4 Evaluation Pilots 

Nineteen EPs with a minimum of a current FAA pilot’s certificate and Class III medical certificate, 
or equivalent, participated. EPs were categorized into pilot types by their experience level. The pilot 
categories and mean total number of flight hours of experience are shown in Table 1. The pilot spectrum 
was selected to enable an evaluation of SVS display technology with application to the diverse GA 
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population. The VFR, IFR, and H-IFR rated pilots represented the spectrum of pilot skills currently 
operating within the National Airspace System (NAS). 

Low-time VFR-rated pilots were included to test the hypothesis that SVS displays can enable them 
to achieve flight technical error, workload and SA ratings that are as good as, or better than, IFR pilots 
flying in IMC using baseline round dials. Instrument-rated pilots with flight time less than 1 ,000 hours 
were included to establish a baseline characterization of minimum IFR-rated performance as well as to 
ascertain the effect that SVS, and other advanced display technologies, would have on this segment of the 
pilot population. Very experienced pilots with instrument ratings and flight time greater than 2,000 hours 
(hereinafter referred to as H-IFR pilots) were also included to determine the effect that SVS and other 
advanced display technologies would have on highly-experienced airline and professional test pilots and 
also to facilitate comparison of results from this study with other studies. All H-IFR pilots were current 
or retired professional pilots. 


Table 1 Evaluation Pilot Data 


Pilot Type 

Number 
of EPs 

Mean Flight 
Time, Hours 

Minimum 
Flight Time, 
Hrs 

Maximum 
Flight Time, 
Hrs 

Standard 
Deviation, Hours 

VFR (<400 hrs) 

6 

241 

102 


107 

IFR (<1000 hrs) 

6 

468 

267 


131 

H-IFR (1000+ hrs) 

7 

4019 

1980 

7000 

1811 

Total 

19 


102 

7000 

2141 


3.5 Scenarios, Training, and Operations 

Flight tests were conducted using Roanoke Woodrum Field in Roanoke, Virginia (FAA identifier 
KROA). It was selected due to the presence of significant terrain and obstructions in its vicinity and 
convenient proximity to NASA Langley Research Center. In particular, Runway 24 was selected since 
terrain and obstacles pose significant operational challenges for approaches to this runway. Runway 24 
was not equipped with visual approach slope indicator lights or a precision approach path indicator. 

Three approach scenarios were created. Two were conventional approaches, and one represented an 
advanced operational concept. The conventional scenarios included a VFR traffic pattern approach in 
VMC and an ILS approach in simulated IMC. The advanced operational concept was an IMC approach 
that used the same flight path flown for the conventional VFR traffic pattern approach; however, this 
“VMC-like” approach was flown in simulated IMC. Figure 8 shows a plan view and side views of the 
evaluation approaches. The three approaches are described in the following sections. 
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Figure 8a: VFR Sectional Map View with Approaches Overlaid 
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Figure 8b: Sideview of VFR and VMC-like Approaches 



Figure 8c: Sideview of ILS approach 
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3. 5. 1 VFR Approach 

The VFR traffic pattern incoiporated a downwind leg displaced approximately 1.4 nm abeam the 
runway at 1,000 ft above the touchdown zone, a base leg, and a 1.5 nm final approach segment. Data 
runs were initiated at 1 00 kts indicated airspeed (IAS) at 1 ,000 ft above the touchdown zone on the 
downwind leg heading at Point A in Figure 8. EPs were instructed to use visual references primarily to 
navigate through this approach. They were instructed to fly parallel to the runway and maintain airspeed 
and altitude with flaps retracted. Once abeam the runway numbers, EPs were to select 1 0 degrees of 
flaps, reduce speed to 90 kts IAS, establish a 1.5 nm final approach, and complete the approach. EPs 
were instructed to continue the approach to 50 ft above ground level (AGL), or to actual touchdown, 
depending on their level of experience and prevailing conditions. The VFR traffic pattern approach was 
included in this evaluation to provide a comparison to current VMC operations. 

3.5.2 ILS Approach 

The ILS approach included a 90-degree base leg, a 30-degree localizer intercept, and a 3.0-degree 
glideslope intercept point approximately nine nm from the runway touchdown markers. This approach 
was performed with restricted visibility through the use of an IFR training visor, which eliminated out- 
the-window visibility. 

Because an ILS was not available for Runway 24 at KROA, Global Positioning System (GPS) 
Wide-Area Augmentation System (WAAS) information was used to calculate and display simulated ILS 
localizer and glideslope data. The altitude above the touchdown zone elevation along with the distance to 
the touchdown zone generated ILS glideslope-like deviation data. Lateral linear flight path error, 
combined with the distance along the flight path to a point 1,000 ft beyond the far (departure end) end of 
Runway 24, was used to generate ILS localizer-like deviation data. Although an ILS localizer is 
nominally scaled for a course width of (full scale fly-left to a full scale fly-right) of 700 feet at the runway 
threshold, the maximum simulated localizer deflection was established as +/-250 ft at the runway 
threshold (+/-2.0 degrees course width). Maximum glideslope deflection was set to +/- 0.7 degrees. 

The ILS approach task started at approximately 2,700 ft above the touchdown zone at 100 kts IAS on 
the base leg heading at Point B in Figure 8. EPs would immediately turn to the intercept heading, once 
the research approach was started at Point B, and intercept the localizer. Once established on the 
localizer, EPs were to select 10 degrees of flaps, reduce speed to 90 kts, intercept the glideslope and 
complete the approach. Pilots were instructed to continue to the decision height (DH) at 200 ft AGL and 
then call out “DH.” At the DH, the safety pilot would say, “Continue to 50 ft AGL,” “Continue to 
landing,” or “Runway not in sight,” depending on pilot skill and prevailing conditions and whether or not 
a simulated missed approach (MA) was required as indicated by the test matrix. Calling out the “DH” 
and potentially initiating a MA was included in the flight test to enhance the realism of the approach and 
to provide data regarding pilots’ ability to identify the “DH” as well as minimize further descent during 
the MA. (These data were analyzed and no effects were observed and are subsequently not reported.) 

The test matrix was designed such that approximately half of the IMC approach runs (ILS and VMC-like) 
included an MA. Each run ended either at touchdown or when positive rate of climb was established on 
the MA. The ILS approach was included to enable data comparisons with current precision approach 
operations. 

3.5.3 VMC-Like Approach 

The VMC-like approach used the same flight path as the VFR approach; however, it was performed 
with GPS navigation and restricted visibility through the use of an IFR training visor. A waypoint 
(“NEWT”) was created for the VMC-like approach on the downwind approach path (see Figure 8) abeam 
the runway numbers to designate the point at which the EPs were to select 1 0 degrees of flaps and 
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decelerate to 90 kts IAS. Localizer and glideslope errors were generated in the same manner as for the 
ILS approach. All EPs flew the VMC-lIke approach with each of the experimental display concepts 
except the BRD. The VMC-like approach had the same termination options as the ILS approach (i.e., 
continue to 50 ft, land, or MA). Each run ended either at touchdown or when positive rate of climb was 
established on the MA. The VMC-like approach represented a potential future operation that could be 
enabled by SVS displays. 

3.5.4 Training 

Evaluation pilots received training prior to data collection to minimize learning effects. Training 
included a one-hour briefing regarding the characteristics of the experiment along with a description of all 
display concepts, evaluation approaches, and subjective and objective measures. Next, EPs received 
approximately 2.5 hours of training using a desktop simulator where all test conditions were practiced. 
Pilots could elect to retry test conditions until they were completely comfortable with all aspects of the 
experiment. In general, EPs did not elect to retry training runs and were able to demonstrate acceptable 
performance. Acceptable performance was defined as GDI deflections within 1.75 dots for the ILS 
approach and controlling the velocity vector to be within approximately 5 degrees combined displacement 
of the guidance cue for the HITS display concepts (i.e., PBG and SVS). For the SCFD display concept, 
acceptable performance was approximately 5 degrees displacement between the guidance cue and 
reference waterline for both pitch and roll. During the portions of the BRD evaluations that did not have 
ILS data (i.e., VFR and initial segment of the ILS approaches) acceptable altitude performance was +/- 
200 ft. Airspeed control within +/-10 kts was acceptable for all training scenarios. In addition, 
acceptable performance included pilot awareness and comfort with the displays, test conditions, and 
approaches. Once the training simulation was completed, EPs received approximately 1.2 hours of flight 
training in the C206 to become familiar with the aircraft, flight-crew protocol, research apparatus and 
scenarios. A specific set of training scenarios was selected to provide EPs with exposure to both types of 
advanced guidance symbologies (i.e., SCFD and PBG), baseline round dials, and both research approach 
flight paths (i.e., the VFR/VMC-like and ILS). Other general airwork was also conducted during the 
flight. No vision-restriction devices were used for the flight training sessions. 

3.5.5 Operations 

For all scenarios, EPs were isolated from actual air traffic control (ATC) communication and asked 
to monitor a simulated ATC channel that would play pre-recorded ATC messages based on aircraft 
location. They answered questions about simulated air traffic and provided other qualitative data (i.e., 
situation awareness, workload, display readability, terrain awareness) immediately after each run. (An 
evaluation of the simulated air traffic responses indicated no usable data to differentiate between 
approaches, pilot types, or display concepts and is not reported herein. Even though pilots were briefed to 
expect questions about ATC instructions to other aircraft, that type of mental recall was found to be very 
different from what pilots are accustomed to and they simply did not recall any ATC instructions for other 
aircraft.) Isolating the EPs from actual ATC communications provided a more uniform workload. For 
the simulated-IMC scenarios, the EPs wore an IFR training visor. No vision restriction was used for the 
VFR approach. 

During each approach, the SPs would establish the aircraft approximately 1.0 nm from the starting 
waypoint (i.e., Waypoint A for the VFR/VMC-like approaches or Waypoint B for the ILS approach in 
Figure 8) on the appropriate heading, altitude, and speed, at which time control would be transferred to 
the EP. The EP would then complete the approach setup and perform the evaluation approach. Once the 
approach was completed, the SP took control of the aircraft and the EP answered the post-run 
questionnaires using an electronic questionnaire tool implemented on a pocket personal computer 
(described in Appendix C). 
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The order of test condition presentation was randomized (subject to some operational efficiencies) to 
further minimize learning, fatigue, and environmental effects. Generally, data collection was performed 
during two approximately 1 .2-hour flights to avoid fatigue, though a few pilots completed the entire test 
matrix in one 2-hour flight. EP participation in the flight test usually spanned two days with training 
occurring on the first day. A substantial effort was made to conduct the formal research evaluations 
within four hours after sunrise in order to reduce atmospheric variability effects within the resulting data. 
Most of the data acquired during this flight test fell within this timeframe. All flight operations were 
conducted in clear day conditions. There were no wind limitations established for the data collection 
operations. 

3.6 Independent Variables 

Several independent variables were used for this test. Display concept, approach, and pilot type 
were explicitly controlled in the test matrix. The test condition (TC) variable was based on combinations 
of approach, display concept and pilot type. A summary listing of the independent variables of the study 
is provided in Table 2. 


Table 2: Independent Variables Used for This Experiment 


# 

Name 

Levels 

1 

Display Concept (4 total) 

BRD, SCFD, PBG, or SVS 

2 

Approach (3 total) 

VFR, 1LS, or VMC-like approaches 

3 

Pilot Type (3 total) 

VFR, IFR, or H-IFR 

4 

Test Condition (13 total) 

VFR approach with BRD 
VFR approach with SCFD 
VFR approach with PBG 
VFR approach with SVS 

1LS approach with BRD for IFR and H-IFR pilots only 

ILS approach with SCFD 

ILS approach with PBG 

ILS approach with SVS 

VMC-like approach with SCFD 

VMC-like approach with PBG 

VMC-like approach with SVS 

ILS approach with SVS for VFR pilots only 

VMC-like approach with SVS 


3.7 Test Matrix 

The test matrix for approach, display concept, visibility, and pilot type is provided in Table 3. Note 
that the BRD display concept was not included in the test matrix for the VMC-like approach because of 
anticipated very poor performance that could involve safety of flight concerns. The poor performance 
was expected as a result of the piloting strategy required for traditional IMC operations with the BRD 
display concept. This method uses selected headings to fly, evaluated for tracking performance over at 
least 10 to 20 seconds before updating to manage lateral error. The relatively rapid pace for the VMC- 
like approach would have been highly challenging for the BRD display concept, especially for the VFR 
pilots. In Table 3, “VMC” indicates that no vision restriction device was used to restrict the EP’s 
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visibility. Similarly, “Sim IMC” indicates that a standard IFR training vision restriction device was used 
by the EPs. 

The VFR and IFR pilots evaluated the BRD display concept for the ILS approach twice while the H- 
IFR pilots evaluated the BRD display concept for the VFR traffic pattern twice. The repeat evaluations of 
these test conditions were included to provide more data for a multivariate discriminant analysis 
technique employed and described in Reference 18. Overall, each EP flew 12 total data-collection runs to 
complete the run matrix (four VFR traffic patterns, four ILS approaches, three VMC-like approaches, 1 
repeat). 


Table 3: Test Matrix 


Ident 

# 

Approach 

Display 

Concept 

Visibility 

Pilot Type 

1 

VFR 

BRD 

VMC 

VFR Pilots 

2 

VFR 

SCFD 

VMC 

3 

VFR 

PBG 

VMC 

4 

VFR 

svs 

VMC 

5 

ILS 

BRD 

Sim IMC 

6 

ILS 

SCFD 

Sim IMC 

7 

ILS 

PBG 

Sim IMC 

8 

ILS 

SVS 

Sim IMC 

9 

VMC-Like 

SCFD 

Sim IMC 

10 

VMC-Like 

PBG 

Sim IMC 

11 

VMC-Like 

SVS 

Sim IMC 

12 

VFR 

BRD 

VMC 

IFR Pilots 

13 

VFR 

SCFD 

| VMC 

14 

VFR 

PBG 

VMC 

15 

VFR 

SVS 

VMC 

16 

ILS 

BRD 

Sim IMC 

17 

ILS 

SCFD 

Sim IMC 

18 

ILS 

PBG 

Sim IMC 

19 

ILS 

SVS 

Sim IMC’ 

20 

VMC-Likc 

SCFD 

Sim IMC 

21 

VMC-Likc 

PBG 

Sim IMC’ 

22 

VMC-Likc 

SVS 

Sim IMC 

23 

VFR 

BRD 

VMC 

H-IFR 

Pilots 

24 

VFR 

SCFD 

VMC 

25 

VFR 

PBG 

| VMC 

26 

VFR 

SVS 

VMC 

27 

ILS 

BRD 

Sim IMC 

28 

ILS 

SCFD 

Sim IMC 

29 

ILS 

PBG 

Sim IMC 

30 

ILS 

SVS 

Sim IMC 

31 

VMC-Likc 

SCFD 

Sim IMC 

32 

VMC-Likc 

PBG 

Sim IMC 

33 

VMC-Likc 

SVS 

Sim IMC 


The TC variable was composed of combinations of specific approaches, display concepts, and pilot 
types, as defined in Table 4. The purpose of the TC variable was to provide explicit comparisons across 
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approaches and provide key reference conditions in the data. One reference condition, TC 5, was defined 
by the IFR and H-IFR pilots flying the 1LS approach with the BRD display concept. This condition 
represents the current standard for operations in IMC. As such, the VFR pilots’ data are not included in 
TC 5. In addition, to draw conclusions regarding the ability of the VFR pilots to pilot an aircraft with 
SVS displays in IMC, TC 12 and TC 13 were created for the ILS and VMC-like approaches, respectively. 
Note that the data for the VFR pilots’ evaluations of the SVS display concept for the ILS and VMC-like 
approaches are also included in TCs 8 and 1 1 . 


Table 4: Definition of Test Condition Variable. 


Test 

Condition 

Approach 

Display 

Concept 

Visibility 

Pilot Types 

1 

VFR 

BRD 

VMC 

VFR, IFR, H-IFR 

2 

VFR 

SCFD 

VMC 

VFR, IFR, H-IFR 

3 

VFR 

PBG 

VMC 

VFR, IFR, H-IFR 

4 

VFR 

SVS 

VMC 

VFR, IFR, H-IFR 

5 

ILS 

BRD 

Sim IMC 

IFR, H-IFR 

6 

ILS 

SCFD 

Sim IMC 

VFR, IFR, H-IFR 

7 

ILS 

PBG 

Sim IMC 

VFR, IFR, H-IFR 

8 

ILS 

SVS 

Sim IMC 

VFR, IFR, H-IFR 

9 

VMC-Like 

SCFD 

Sim IMC 

VFR, IFR, H-IFR 

10 

VMC-Like 

PBG 

Sim IMC 

VFR, IFR, H-IFR 

11 

VMC-Like 

SVS 

Sim IMC 

VFR, IFR, H-IFR 

12 

ILS 

SVS 

Sim IMC 

VFR 

13 

VMC-Like 

SVS 

Sim IMC 

VFR 


3.8 Dependent Measures 

Quantitative and qualitative dependent measures were collected in this study. These measures are 
explained below. 

3.8.1 Qualitative Measures 

Post-run questionnaire results analyzed in this report consisted of the 3 -dimensional Situation 
Awareness Rating Technique (SART, see Reference 19), 6-dimensional Task Load Index (TLX, see 
Reference 20), Display Readability Rating (DRR, see Reference 21) and Terrain Awareness (TA) 
question. All of the post-run questionnaires are included in Appendix C. The SART questionnaire was 
applied to measure the EP’s perceived ability to understand the current situation (i.e., situation 
awareness). All references to SA herein are from SART results. 

The SART results are calculated by adding the results from the responses of the perceived supply of 
attentional resources and understanding of the situation questions and then subtracting the response from 
the demand on attentional resources question. However, an unconventional implementation of SART was 
employed using a 0 to 100 scale rather than the nominal 7-pt Likert scale. In this implementation, the 
three SART components (supply, understanding, and demand) were rated 0 to 100 by each pilot with 1 00 
representing “high” and a score of 0 representing “low.” As such, the SART scores can vary from a 
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maximum of 200 to -100. A SART score of 200 would indicate that a pilot had the maximum perceived 
supply of attentional resources and understanding of the situation while also experiencing virtually no 
demand on attentional resources. A SART score of -100 would indicate the pilot had a maximum 
perceived demand on attentional resources and the least possible supply of attentional resources and 
understanding of the situation. 

Pilot subjective workload was assessed through use of the TLX scale. Pair-comparisons of the TLX 
factors were not administered to the EPs to obtain an individualized workload metric; instead, the TLX 
results were calculated by determining the average response to the six questions: 1) Mental demand; 2) 
Physical demand; 3) Temporal demand; 4) Performance; 5) Effort; 6) Frustration level. See Appendix C 
for details. TLX results can vary from a maximum of 100, that would indicate the highest (worst) results 
for all 6 questions, to a minimum of zero (best), which would result from the lowest response for all 6 
questions. 

The DRR scale was applied to assess the workload associated with obtaining information from the 
various research displays. Pilots were provided a definition of desired and adequate performance for the 
DRR scale and self-rated their own performance. Ratings are generated through the use of a decision 
tree. It requires the EPs to answer yes or no to series of questions that ultimately create a phrase 
association that best describes the pilot’s perception of a specific condition in the test matrix. The DRR 
ratings begin from a maximum (worst) of 1 0 which would indicate that “mandatory improvements to the 
display are required” combined with the statement that “the display symbology cannot be used for the 
required task”. The best rating, a DRR rating of 1, would indicate that the display was “satisfactory 
without improvement” combined with the statement that “pilot compensation was not a factor for desired 
performance”. The DRR is a slightly different form of pilot workload measure than TLX, and attempts to 
assess the amount of mental workload expended to absorb flight information from the display device to 
meet desired and/or adequate performance. In this sense, “absorb” means to comprehend the information 
being supplied and transform that information into useable situation awareness and cognitive processing. 

A specific question regarding terrain awareness was also included to acquire data specific to this 
element of SA. The TA results was based on the EP’s overall understanding of the terrain environment 
they were operating within. Terrain Awareness results could vary from a maximum (best) of 100, that 
would indicate that the EP had the best perceived understanding of the terrain environment, down to a 
minimum of zero (worst) that would indicate the least possible terrain awareness. 

3.8.2 Quantitative Measures 

Quantitative data recorded included flight path and airspeed errors. Parameters included: linear 
flight path errors (lateral and vertical in ft), angular flight path errors (localizer and glideslope in dots of 
deflection), and indicated airspeed errors (in knots). Quantitative error parameters are discussed for 
combined Segments 2 and 3 as well as Segment 3 alone (see Figure 8 for definitions of Segments 2 and 
3). Indicated airspeed errors are discussed separately for Segments 2 and 3 due to data analysis 
limitations. Root-mean-square (RMS) values of these error variables are included in the analyses 
performed and reported herein. 

Linear lateral and vertical flight path and track-angle errors were calculated using a technique that 
determined the shortest lateral distance from the aircraft to the flight path centerline. The intersection of 
this line with the desired flight path was then used to establish reference conditions to generate error 
parameters. The linear errors were combined with range to the runway touchdown point to calculate 
glideslope errors, and the range to a point 1 ,000 ft beyond the far (departure end) end of Runway 24 for 
localizer errors. Proceeding in this manner enabled localizer and glideslope errors to be calculated, 
displayed, and analyzed the 1LS approach and calculated and analyzed for the VFR and VMC-like 
approaches. Note that the localizer and glideslope error was only displayed for the ILS approach. 
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For the simulated-IMC approaches (i.e., ILS and VMC-like) the FAA IFR Practical Test Standards 
(PTS) were applied. To earn an instrument rating, pilots must demonstrate the ability to maintain flight 
path and airspeed errors within criteria outlined in the PTS. The accepted and desired IMC performance 
conditions were based on the FAA PTS criteria defined in Reference 22. The resulting “IFR PTS 
boundary” used for both the ILS and the VMC-like approaches began at a specified entry gate and ended 
at the DH. It was bounded by 3 /4-scale maximum deflections (1.5 dots) of localizer and glideslope 
indicators, and +/-10 kts of IAS error. Note that the instantaneous required airspeed change during the 
glideslope intercept from 100 +/-10 kts to 90 kts to +/-10 kts resulted in a potential limitation of the 
maximum amount of time pilots’ could achieve PTS performance. If a pilot was within the PTS IAS limit 
during the localizer tracking segment, but with an IAS above 100 kts, then at the point of glideslope 
intercept he would no longer be within the PTS criteria until he reduced IAS to be below 100 kts. It is 
considered that this momentary IAS exceedance would be acceptable performance during an IFR 
checkride. However, for the analysis performed herein the amount of time required to bring IAS back to 
within +/-10 kts error detracts from the time within PTS criteria. This artifact would also apply to the 
VMC-Like approaches since a speed change is required for that maneuver too. While exceeding the IFR 
PTS does not directly imply an imminent accident, boundary violation does provide an indication of an 
unstable approach and an increased accident risk or exposure. PTS results for all three approaches are 
discussed as the percentage of time the evaluation pilots were able to satisfy all three parameters 
simultaneously. 

Linear lateral flight path error analysis for the various display concepts and approaches was valuable 
in support of RNP-style approaches that are currently being developed and used within the NAS. For 
RNP operations, the aircraft, avionics, and crew must be certified by the FAA to be capable of remaining 
within linear lateral limits of the centerline of the desired flight path 95% of the time. The required total 
system performance can be achieved by various combinations of flight management systems, navigation 
sensors, flight guidance systems, cockpit displays and other equipment, together with monitoring and 
alerting capabilities and additional training, which distinguishes performance-based navigation from other 
navigation methods. RNP aircraft certification typically specifies an accuracy level, such as 0.10 nm or 
0.30 nm, at which the aircraft can navigate (Reference 23). This is the combination of airframe and 
autoflight system for coupled operations. For manual operations, the crew is allocated part of the total 
system error budget. This is different compared to angularly-defined airspace limits consistent with most 
of the traditional IMC approach operations. For this report, a flight technical error criterion of 0. 1 nm is 
used as a reference to compare and contrast results from the different display concepts, approaches, and 
pilot types for manually flown approaches. The results discussed herein only deal with the ability of the 
pilot to interact with the display concepts and minimize flight path error (i.e., only accounting for what is 
known as flight technical error) and does not account for errors within the navigational system (e.g., 

GPS). Dynamic taxi testing along the runway centerline at KROA showed that the installed Air Data, 
Attitude, and Heading Reference System (ADAHRS) equipment performance exhibited errors on the 
order of 10 to 20 ft (i.e., 0.0016 to 0.0033 nm). These errors were considered negligible for the in-flight 
results discussed herein 

For the ILS Approach, Segment 3 extended from the glideslope intercept point at 8.4 nm and ended 
at the decision height (i.e., 200 ft AGL, approximately 0.66 nm from the glideslope intercept point). For 
the advanced display concepts, guidance was provided throughout the entire ILS approach. However, for 
the BRD display concept, evaluation pilots flew a specified heading and altitude until intercepting the 
localizer from the start point, then flew a specified altitude while tracking the localizer to intercept the 
glideslope. The ILS approach was completed with the EPs following the localizer and glideslope signals 
to decision height. While all ILS evaluations were initiated from the same location, the guidance for the 
BRD display concept was not consistent for the entire approach and did not have positive lateral and 
vertical guidance throughout the localizer intercept and localizer tracking phases of the approach 
(Segment 2). To compensate for the inconsistent guidance of the BRD display concept, comparisons of 
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the flight path error data from the BRD display concept were limited to the localizer and glideslope 
tracking phases of the 1LS approach (Segment 3). 

If, during a run, the localizer and glideslope indicators were simultaneously at their upper limits and 
the path error was increasing, the EP was deemed to have lost all flight path awareness and the TE would 
terminate the run. The SP was prepared to take control immediately in any situation involving safety-of- 
flight concerns; however, no runs were aborted by the SP due to safety-of-flight issues during the test. 


4.0 Data Analyses and Results 

Results are presented in this section for subjective situation awareness, workload, display readability, 
and terrain awareness. Quantitative results include percent of time within the PTS, localizer errors, 
glideslope errors, indicated airspeed errors, lateral path errors, and vertical path errors,. The advanced 
display concepts provide guidance to minimize linear flight path errors. In order to best compare pilot 
performance when using the advanced display concepts linear path errors analyses are used. Linear 
pathway error analyses also provides the ability to assess performance compared to RNP criteria that 
could be associated with future advanced landing approaches. However, linear path error analyses 
adversely penalize displays that minimize angular pathway errors, such as the BRD with the localizer and 
glideslope errors. For this report both linear and angular pathway errors are employed. PTS criteria are 
employed for the localizer, glideslope and IAS error analyses. Required Navigation Performance criteria 
of +/-0.1 nm and +/-100 ft are used for the linear lateral and vertical error analyses, respectively. 

For all parameters, analyses are performed separately for each approach (i.e., VFR, ILS, and VMC- 
like approaches) using all of the pilot group’s data. This enables a clear view of the display concepts and 
pilot types without considering operational task (i.e., approach types) effects. Then, in an effort to 
describe the results within a framework aligned with current operations, the VFR pilots’ data are removed 
from the ILS approach. To provide increased statistical power to identify effects of display concept, 
approach, and pilot type, data for the IMC approaches (i.e., ILS and VMC-like approaches) are analyzed 
together. To provide a direct comparison of the advanced display concepts and approaches with current 
IMC and VMC operations, the test condition analyses are provided. Recall that the test condition 
independent variable is a combination of approach and display concept (i.e., VFR approach + BRD, etc.). 

During the course of the flight test experiment, one run had to be terminated due to the pilot losing 
complete awareness of the desired flight path. The test condition for this run was for a VFR pilot flying 
the ILS approach with the BRD display concept. For this ILS approach run, the pilot was able to 
maneuver near the desired flight path for several minutes and make some progress through the localizer 
intercept and localizer tracking phase of the approach. When the added workload of intercepting and 
tracking the glideslope was added, the pilot was apparently overloaded and never began the descent. The 
run was terminated at that time. The pilot had difficulties from the beginning of the approach with 
consistently poor performance up to the point of termination at approximately 2.5 nm from touchdown. 
The results were consistent throughout this ILS approach attempt and were included in the data. 

Analyses of variance (ANOVAs) were conducted using the SPSS™ software package to test for 
statistical significance among the independent variables for the different measures. Results due to the 
independent variables are referred to as main effects and are differentiated from those resulting from 
interactions between the independent variables. In cases where the statistical significance factor (p) was 
less than 0.05, the independent variable differences were considered to be significant for the measure 
being evaluated. For cases where “p” was less than 0.01, differences were considered to be highly 
significant. Student-Newman-Keuls (SNK) post-hoc analysis results were used when “p” was less than 
0.05 and there were more than 2 items being compared. SNK analysis provides subsets of the results 
based on their statistical relationship to each other. Results in the same subset are considered to not be 
statistically different from others in that subset. Those in different subsets are considered to be 
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statistically different. For the ordinal DRR data, a descriptive analysis of the medians, maximums, 
minimums, and percentiles was performed. 

4,1 Situation Awareness 

SART results for the four display concepts and three approaches are presented in Figure 9 along with 
Table 5. Individual SART results for each evaluation are provided in Appendix D. 


Table 5: SART Descriptive Statistics (all Pilots) 


Display 

Concept 

Approach 

Mean 

Standard 

Deviation 

Number of 
Data Points 

BRD 

VFR 

104 

54.3 

24 


1LS 

32.6 

49.9 

29 


Total 

67.5 

60.1 

53 

SCFD 

VFR 

78.2 

54.1 

19 


1LS 

75.8 

33.9 

18 


VMC-Like 

62.4 

55.0 

18 


Total 

72.3 

48.4 

55 

PBG 

VFR 

115 

45.6 

18 


1LS 

83.1 

46.6 

17 


VMC-Like 

87.6 

46.5 

19 


Total 

95.3 

47.5 

54 

svs 

VFR 

113 

48.5 

19 


ILS 

114 

53.6 

21 


VMC-Like 

108 

44.8 

19 


Total 

112 

48.5 

59 

Total 

VFR 

101 

52.0 

75 


ILS 

78.5 

54.0 

74 


VMC-Like 

86.6 

51.5 

56 


Total 

88.8 

53.2 

205 
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Figure 9: SART Results, All Display Concepts, All Pilots and All Approaches 


4.1.1 VFR approach 

For the VFR approaches, EPs were instructed to navigate the aircraft using outside visual cues and to 
cross check with the guidance when it was provided by the advanced displays. An ANOVA conducted on 
the SART data for this approach for the main effects of display concept and pilot type resulted in no 
significant (p>0.05) results. VFR evaluations of the PBG and SVS display concepts showed trends 
toward increased SA, though not significant, compared to the BRD results. The pilots’ favorable SA 
ratings were most likely due to the presence of the PUTS symbology, even though they were primarily 
performing a VFR task. Evaluation pilots were able to cross check their VFR-style navigation with the 
FIITS information and easily confirm their position. In the case of the SCFD display concept, the 
evaluation pilots were unable to easily confirm their relative position with respect to the desired path 
since flight director commands are a function of both track-angle and lateral offset errors. As a result, a 
given roll command did not provide intuitive direct indication of how far pilots were away from the 
desired flight path, leading to poorer ratings for the SCFD for the VFR approach compared to the other 
display concepts, including the BRD. 

4.1.2 ILS approach, all pilots 

An ANOVA conducted on the SART results for the main effects of pilot type and display concept 
for the ILS approach revealed that display concept (F(3,85)=22.4, p<0.01) was highly significant but was 
not significant for pilot type (p>0.05). There were no significant interactions between the main effects. 

A Student-Newman-Keuls (SNK) post-hoc analysis on the SART ratings resulted in the three unique 
subsets shown below in Table 6. The BRD display concept was rated as providing the least SA of the 
four display concepts tested. The middle subset was comprised of the SCFD and the PBG display 
concepts. The SCFD and PBG display concepts were rated as providing improved SA compared to the 
BRD display concept but there were no significant differences between these two display concepts. The 
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significant increase in SA for the SCFD and PBG display concepts compared to the baseline round-dial 
concept is most likely due to the presence of the moving map ND and/or integrated PFD compared to the 
separate gauges of the baseline round-dial display concept. The SVS display concept was rated as 
providing the highest SA among the concepts tested. The significant difference in SA between the SVS 
and the PBG display concepts can be attributed to the addition of SVS terrain since this was the only 
difference between these two concepts. 

Situation Awareness results for the SVS concept used to fly the 1LS approach were higher than those 
for the BRD display concept for the ILS approach. This result may reflect the intuitive integration of the 
outside environment, as provided by SVS terrain, with the desired flight path and aircraft state data. 

Table 7 provides SART results for the three pilot groups for the ILS approach for each of the display 
concepts along with the mean for all display concepts. These data are presented to show that SA from the 
three pilots groups had some variability, although it was not significant. Overall, the H-IFR pilot group 
reported higher SA than the other two pilot groups, which may be attributed to the H-IFR pilots' order-of- 
magnitude higher amount of flying time, as presented in Table 1. IFR pilots, however, reported lower SA 
than the VFR pilots for three of the four display concepts evaluated, even though they had received the 
training required to acquire an IFR rating. An analysis of the constituent elements of SART for all 
display concepts for the ILS approach revealed that the VFR pilots reported significantly higher results 
(F(2,85)=4.3, p=0.02) for the 3 rd SA question (understanding of the situation) for the BRD display 
concept than the IFR pilots and were included in the subset with the H-IFR pilots. There were no other 
significant differences for the other SART questions for all the other display concepts (i.e., p>0.05). 

While the VFR pilots had not acquired the training to become IFR pilots, for all of the display concepts 
they reported SA results that were an average of 10 SA units higher than the IFR pilots as a group (but not 
statistically significant). All pilot groups exhibited similar trends in their SART results for the different 
display concepts with the BRD being the worst and SVS display the best and the SCFD and PBG display 
concepts in between. The bias in the VFR pilots’ SART results needs to be considered when drawing 
conclusions that compare pilot groups. An example of comparisons across pilot groups is provided 
during the test condition analysis shown in this report. 


Table 6: Mean SART SNK Results for all Pilots, ILS Approach 


Display 

Concept 

Number of 
Data Points 

Subset 

1 

2 

3 

BRD 

29 

32.6 



SCFD 

18 


75.8 


PBG 

17 


83.1 


SVS 

21 



113.5 

Significance 


1.0 

0.5 

1.0 


Table 7: Mean SART SNK Results for All Pilots for All Display Concepts, ILS Approach 


EP-type 

Number 
of data 
points 

SART 

SART 

BRD 

SART 

SCFD 

SART 

PBG 

SART 

SVS 

IFR 

30 

58.7 

14.1 

69.8 

76.4 

100.5 

VFR 

29 

69.2 

32.3 

66.0 

95.0 

114.2 

H-IFR 

26 

90.1 

62.3 

91.7 

76.8 

127.9 

Significance 


0.26 
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4.1.3 VMC-Like approach 

An analysis of the VMC-like approach data was performed to determine the effect of display concept 
and pilot type for this advanced IMC approach. An ANOVA revealed that display concept was highly 
significant (F(2,56)=5.5, p<0.01) for SART ratings but pilot type was not significant (p>0.05) There 
were no significant interactions between display concept and pilot type. An SNK post-hoc analysis on 
SA resulted in two overlapping subsets as shown in Table 8. The SVS display provided significantly 
more SA than the SCFD but there were no differences between the PBG and SVS displays. The SA 
improvements with the SVS display concept over the SCFD were most likely due to the combination of 
FIITS guidance symbology and presence of SVS terrain. 

Table 8: Mean SART SNK Results for All Pilots for All Display Concepts, VMC-Like Approach 


Display 

Concept 

Number of 
Data Points 

Subset 

1 

2 

SCFD 

18 

62.4 


PBG 

19 

87.6 

87.6 

SVS 

19 


108.4 

Significance 


0.08 

0.15 


4.1.4 ILS approach, without VFR pilot data 

An analysis of the ILS approach data without the low-time VFR pilots’ data was performed to create 
a baseline reflecting current operations. An ANOVA conducted on SA for the IFR and Fl-IFR pilot data 
revealed that the main effect on SA was statistically significant (F(l,56)=6.247, p<0.017) for pilot type 
and highly significant (F(3,56)=12.475, p<0.01) for display concept. There were no significant 
interactions between pilot type and display concept. The IFR-rated pilots reported significantly lower SA 
than the high-time pilots for all display concepts as shown in Table 9. This result may reflect the total 
number of flight hours for the high-time pilots, their overall level of skill with the BRD display concept, 
and also their ability to adapt to the advanced display concepts, resulting in greater overall situation 
awareness. An SNK post-hoc analysis on the SA revealed three unique subsets for display concept as 
shown below in Table 10. The BRD display concept provided the pilots with significantly lower SA than 
all other concepts tested as indicated in Figure 9. The SCFD and PBG display concepts were rated as 
providing significantly better SA than the BRD display concept, but there were no significant differences 
among the SCFD and PBG display concepts. The SVS concept provided the pilots with significantly 
higher SA than all other concepts tested, and this result was most likely due to the presence of the SVS 
terrain. 


Table 9: Mean SART results for IFR and H-IFR Pilots for the ILS Approach 


Pilot 

Type 

Number of 
Data Points 

Mean 

IFR 

H-IFR 

30 

63.7 

26 

90.9 
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Table 10: Mean SART SNK Results for Display Concept for IFR and H-IFR Pilots for the ILS 

Approach 


Display 

Concept 

Number of 
Data Points 

Subset 

1 

2 

3 

BRD 

18 

32.8 



PBG 

11 


76.6 


SCFD 

12 


80.8 


SVS 

15 



113.3 

Significance 


1.000 

0.77 

1.000 


4.1.5 ILS and VMC-Like approach analysis, all pilots, advanced displays 

For the VMC-like approach, no data were acquired for the BRD display concept, as discussed 
previously. An ANOVA was conducted on the main factors of advanced display concepts (PBG, SCFD, 
SVS) and approach (ILS, VMC-like) for the SART ratings. Results indicate that display concept was 
highly significant (F(2,l 12)=1 0.85 1 , p<0.01) and approach was not significant (p>0.05). Post-hoc 
analysis, shown in Table 11, revealed two unique subsets for the SA results. As shown in Figure 9, the 
SVS display provided significantly better SA than the PBG display due to the addition of terrain on the 
SVS display. Recall that the only difference between the SVS and PBG display concepts was the SVS 
terrain. There were no significant SA differences between the PBG and SCFD concepts. 

Overall, the effect of ILS or VMC-like approach was fairly small for the PBG and SVS display 
concepts as compared to the SCFD display concept as shown in Figure 9. For the VMC-like approach, 
SART results for the SCFD, PBG, and SVS display concepts were as good or better than those for the 
BRD display concept for the ILS approach without the VFR pilots’ data. In addition, SA results for the 
SVS display concept as shown in Figure 9 were similar for all evaluations (i.e., VFR, ILS and VMC-like). 

Table 11: Mean SART SNK Results for Advanced Display Concepts for All Pilots for ILS and 

VMC-Like Approach 


Display 

Concept 

Number of 
Data Points 

Subset 

1 

2 

SCFD 

36 

69.1 


PBG 

36 

85.5 


SVS 

40 


111.1 

Significance 


0.086 

1.000 


4.1.6 Test condition analysis 

A subsequent analysis of the SART data was performed based on test condition (TC), with post-hoc 
results presented in Table 12. An ANOVA performed on the SART data indicated that TC was highly 
significant (F(12,221)=5.585, p<0.01). In Table 12, three subsets were identified. TC 5, the ILS 
approach flown with the BRD concept, was rated as providing significantly less SA than the VFR 
evaluations of all display concepts. The SVS concepts (regardless of which approach was flown) were 
rated as providing the highest SA and were not found to be statistically different from TC 1 (i.e., VFR 
evaluation of the BRD display concept). The VMC-like evaluation of the SCFD display concept (TC 9) 
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was rated as providing significantly less SA for the VMC-like approach than the VFR evaluations of the 
other display concepts. 


Table 12: Mean SART SNK Results for Test Condition 


Test Condition Name 

Test 

Condition 

Number of 
Data Points 

Subset 

1 

2 

3 

1LS Approach + BRD (no VFR pilots) 

5 

18 

32.8 



VMC-like Approach + SCFD 

9 

18 

62.4 

62.4 


1LS Approach + SCFD 

6 

18 


75.8 

75.8 

VFR Approach + SCFD 

2 

19 


78.2 

78.2 

ILS Approach + PBG 

7 

17 


83.1 

83.1 

VMC-like Approach + PBG 

10 

19 


87.6 

87.6 

VFR Approach + BRD 

1 

24 


104.0 

104.0 

VMC-like Approach + SVS 

11 

19 


108.4 

108.4 

VFR Approach + SVS 

4 

19 



112.6 

ILS Approach + SVS 

8 

21 



113.5 

ILS Approach + SVS (only VFR pilots) 

12 

6 



114.2 

VFR Approach + PBG 

3 

18 



114.9 

VMC-like Approach + SVS (only VFR pilots) 

13 

6 



115.5 

Significance 



0.064 

0.065 

0.315 
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4.2 Subjective Workload 

Subjective workload results for the four display concepts and three approaches for all pilots are 
presented in Figure 10 and Table 13 and were generated through use of the TLX questionnaire and 
calculation process described in Appendix C. (Error bars in Figure 10 represent one Standard Deviation 
(SD).) Individual TLX results for each evaluation are provided in Appendix D. 


Table 13: TLX Descriptive Statistics (all Pilots) 


Display 

Concept 

Approach 

Mean 

Standard 

Deviation 

Number 
of Data 
Points 

BRD 

VFR 

27.2 

14.1 

24 


1LS 

46.2 

20.4 

29 


Total 

36.1 

19.1 

53 

SCFD 

VFR 

36.4 

16.1 

19 


1LS 

36.1 

17.1 

18 


VMC-Like 

39.3 

19.4 

18 


Total 

37.3 

17.3 

55 

PBG 

VFR 

27.3 

12.4 

18 


ILS 

28.9 

16.0 

17 


VMC-Like 

34.8 

15.9 

19 


Total 

30.5 

14.9 

54 

svs 

VFR 

27.8 

14.8 

19 


ILS 

25.4 

15.7 

21 


VMC-Like 

29.7 

14.4 

19 


Total 

27.6 

14.9 

59 

Total 

VFR 

29.7 

14.9 

75 


ILS 

33.8 

18.3 

74 


VMC-Like 

34.5 

16.8 

56 


Total 

33.1 

17.3 

221 
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Figure 10: TLX Results, All Display Concepts, All Pilots, and All Approaches 

4.2.1 VFR approach 

Results for all of the display concepts indicated low workload for the VFR approach. This result was 
expected since the VFR approach was conducted without the vision restriction device in clear day VMC. 
An ANOVA conducted on the TLX results for the VFR approach for the main effects of display concept 
and pilot type resulted in no significant (p>0.05) results. 

4.2.2 ILS approach, all pilots 

An ANOVA conducted on the TLX results for all pilots and display concepts for the ILS approach 
revealed that the main effect was highly significant for display concept (F(3,85)=18.3, p<0.01) but not for 
pilot type (p>0.05). A SNK post-hoc analysis on the TLX data resulted in the three unique subsets for 
display concept shown below in Table 14. For the ILS approach for all pilots, workload was significantly 
decreased by 10 units for the SCFD display concept compared to the BRD display concept most likely 
reflecting the improvements due to the moving map and the integrated single-cue flight director on the 
PFD display. Workload was also significantly reduced by 7.2 units for the PBG display concept 
compared to the SCFD display concept reflecting the intuitive guidance provided by the HITS and flight 
path guidance symbology. Incorporation of the terrain on the SVS display concept provided a small 3.5 
unit decrease in workload as compared to the PBG display concept. 
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Table 14: Mean TLX SNK Results for Display Concept for all Pilots and Display Concepts, ILS 

Approach 


Display 

concept 

Number of 
Data Points 

Subset 

1 

2 

3 

SVS 

21 

25.4 



PBG 

17 

28.9 



SCFD 

18 


36.1 


BRD 

29 



46.2 

Significance 


0.302 

1.000 

1.000 


4.2.3 VMC-Like approach 

An analysis of the VMC-lIke approach data was performed to determine the effect of display concept 
and pilot type for this advanced IMC approach. An ANOVA conducted on the TLX results for all pilots 
and display concepts revealed that the main effect was highly significant for display concept 
(F(2,56)=5.5, p<0.01) and also for pilot type (F(2,56)=8.5, p<0.01). There were no significant 
interactions between the main effects. An SNK post-hoc analysis on the TLX results revealed two 
overlapping subsets for display concept and two unique subsets for pilot type, as shown in Tables 15 and 
16. While the PBG guidance symbology decreased workload by approximately 4.5 units compared to the 
SCFD for this approach, this improvement was not statistically significant. The effect of SVS terrain also 
decreased workload compared to the PBG display concept by a similar amount. The decreased workload 
for the SVS display, due to the combination of TUTS guidance symbology and SVS terrain, created a 
significant 9.4 unit reduction compared to the SCFD display concept. This can be seen in Table 15, 
which shows that the SVS display concept was in the lower (best) workload subset while the SCFD 
display concept was in the higher (worst) subset. The PBG display concept was similar to both and 
included in each subset. It is believed that the higher demand of the VMC-like approach contributed to 
the significant results for pilot type as compared to the ILS approach. Overall, the VFR pilots provided 
significantly lower TLX results than the IFR and H-IFR pilots. It is speculated that the VFR pilots 
perceived the workload of the VMC-like approach flown with the advanced display concepts to be closer 
to the VFR approach, which is the only approach they are accustomed to flying. 

Table 15: Mean TLX SNK Results for the advanced Display Concepts for the VMC-Like 

Approach 


Display 

Concept 

Number of 
Data 
Points 

Subset 

1 

2 

SVS 

19 

29.7 


PBG 

19 

34.8 

34.8 

SCFD 

18 


39.3 

Significance 


0.075 

0.112 
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Table 16: Mean TLX SNK Results for Pilot type for the VMC-Like Approach 


Pilot Type 

Number of 
Data Points 

Subset 

1 

2 

VFR 

18 

28.1 


H-IFR 

20 


35.8 

IFR 

18 


39.5 

Significance 


1.000 

0.192 


4.2.4 ILS approach without VFR pilot data 

Results from an ANOVA conducted on the TLX data without the VFR pilot data indicated that pilot 
type (F(l,56)=5.0, p=0.032) was significant and display concept (F(3,56)=6.19, p<0.01) was highly 
significant for TLX results for this analysis. The IFR pilots reported significantly higher workload (7 unit 
increase) than the FLIFR pilots. This result reflects the total flight hours for the high-time pilots, their 
overall level of skill with the BRD display concept, and possibly their ability to adapt to the advanced 
display concepts, resulting in lower overall subjective workload. An SNK post-hoc analysis on the TLX 
data for display concept resulted in the two overlapping subsets shown in Table 17. The BRD display 
concept was rated as providing significantly higher workload than the SVS and PBG concepts, but it was 
not significantly different from the SCFD concept. The SCFD was rated as having no significant 
workload differences from any of the other display concepts. Pilots appear to more easily follow the 
HITS pathway guidance. While not significant, the addition of the SVS terrain lowered subjective 
workload four units compared to the PBG display concept while also significantly improving SA as noted 
above. 

Table 17: Mean TLX SNK Results for Display Concept for IFR and H-IFR Pilots for all Display 

Concepts for the ILS approach 


Display 

Concept 

Number of 
Data 
Points 

Subset 

1 

2 

SVS 

15 

27.6 


PBG 

11 

31.6 


SCFD 

12 

36.4 

36.4 

BRD 

18 


44.8 

Significance 


0.15 

0.073 


4.2.5 ILS and VMC-Like approach analysis, all pilots, advanced displays 

For the VMC-like approach, no data were acquired for the BRD display concept as discussed 
previously. An ANOVA was conducted on the data for the three advanced display concepts for the ILS 
and VMC-like approach for all pilot types. Results indicated that display concept was highly significant 
(F(2,112)=8.558, p<0.01) and approach was significant (F (1,1 12)=5.544, p=0.021) for the workload 
ratings. The effect of display concept and approach are shown in Figure 10, where the SVS display 
provides the lowest workload for both approaches and the workload for the VMC-like approach increases 
for all of the advanced display concepts. Results from the SNK post-hoc analysis are presented in Table 
18. Both the PBG and SVS display concepts significantly reduced workload compared to the SCFD 
display concept, most likely due to the HITS pathway guidance. The workload for the SVS display 
concept was reduced by 4.5 units compared to the PBG display concept. The VMC-like approach 
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increased workload compared to the 1LS approach for all advanced displays by 3.2, 5.9, and 4.3 units for 
the SCFD, PBG, and SYS displays, respectively, as shown in Table 13 and Figure 10. 


Table 18: Mean SNK TLX Results for Display Concept for all Pilots for the ILS and VMC-Like 

Approach (Advanced Display Concepts) 


Display 

Concept 

Number of 
Data Points 

Subset 

1 

2 

SVS 

40 

27.5 


PBG 

36 

32.0 


SCFD 

36 


37.7 

Significance 


0.068 

1.000 


4.2.6 Test condition analysis 

An analysis was performed based on TC with post-hoc results presented in Table 19. Test Condition 
was highly significant for TLX results (F(12,221)=4.172, p<0.01). Four overlapping subsets were formed 
from post-hoc TC analysis. The VFR evaluations of BRD, PBG, and SVS display concepts, the ILS 
evaluations of the PBG and SVS display concepts and the VMC-like evaluation of the SVS display 
concept were rated as providing the lowest workload and contained in subset 1. The ILS approach with 
the BRD display concept (TC 5) was rated as requiring the highest workload. Test Condition 5 grouped 
with the VMC-Like approach evaluations of the PBG and SCFD display concepts, the ILS approach 
evaluation of the SCFD display concept, and the VFR approach evaluation of the SCFD display concept. 
Workload ratings for all of the TCs in the highest workload subset were significantly higher than those in 
the lowest workload subset. These results indicate that for the ILS approach the SVS and PBG display 
concepts (but not the SCFD concept) offer significantly reduced workload compared to the BRD display 
concept. Only the SVS display concept provides significantly reduced workload for the VMC-like 
approach compared to the BRD display concept for the ILS approach (TC 5). For the ILS approach, the 
SVS display concept provides workload as low or lower than for the VFR approach with the BRD display 
concept (TC 1). The advanced display concepts may potentially enable advanced IMC operations, such 
as the VMC-like approach, since their workload results were less than the ILS approach with BRD 
display concept (TC 5). 
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Table 19: Mean TLX SNK Results for Test Condition 


Test Condition Name 

TC 

# 

Number of 
Data Points 

Subset 

1 

2 

3 

4 

ILS Approach + SVS (only VFR pilots) 

12 

6 

19.8 




VMC-Like Approach + SVS (only VFR pilots) 

13 

6 

23.7 

23.7 



ILS Approach + SVS 

8 

21 

25.4 

25.4 



VFR Approach + BRD 

1 

24 

27.2 

27.2 

27.2 


VFR Approach + PBG 

3 

18 

27.3 

27.3 

27.3 


VFR Approach + SVS 

4 

19 

27.8 

27.8 

27.8 


ILS Approach + PBG 

7 

17 

28.9 

28.9 

28.9 


VMC-Like Approach + SVS 

11 

19 

29.7 

29.7 

29.7 


VMC-Like Approach + PBG 

10 

19 


34.8 

34.8 

34.8 

ILS Approach + SCFD 

6 

18 


36.1 

36.1 

36.1 

VFR Approach + SCFD 

2 

19 


36.4 

36.4 

36.4 

VMC-Like Approach + SCFD 

9 

18 



39.3 

39.3 

ILS Approach + BRD (no VFR pilots) 

5 

18 




44.8 

Significance 



0.2 

0.1 

0.1 

0.1 


4.3 Display Readability Rating 

Display readability rating (DRR) results for the four display concepts and three approaches are 
presented in Figure 1 1 and Table 20 and were generated through use of the DRR questionnaire in 
Appendix C. Table 20 presents the percentage of the time that pilots selected a specific DRR rating for 
the combinations of display concept and approach. The median and total number of data points are also 
given. Figure 1 1 presents the cumulative percentage that pilots selected specific DRR ratings for each 
display concept for the VFR, ILS, and VMC-Like approaches. Discussion is provided regarding the 
minimum, maximum, and median ratings received for each display concept and test condition. Further 
discussion is provided regarding the range of ratings required to capture at least 60% and 80% of the 
ratings to provide insight into the distribution of the results. Individual DRR results for each evaluation 
are provided in Appendix D. 
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Table 20: Display Readability Percentage of Time Selected Results (all Pilots) 


DRR 

VFR Ap 

proach 

1LS Approach 

VMC-L Approach 

BRD 

SCFD 

PBG 

SVS 

BRD 

SCFD 

PBG 

SVS 

SCFD 

PBG 

SVS 

1 

25.0 

15.8 

27.8 

45.0 

10.3 

22.2 

17.6 

57.1 

21.1 

26.3 

41.2 

2 

33.3 

31.6 

44.4 

35.0 

20.7 

50.0 

47.1 

28.6 

42.1 

42.1 

35.3 

3 

29.2 

21.1 

22.2 

10.0 

20.7 

11.1 

29.4 

9.5 

5.3 

15.8 

5.9 

4 

8.3 

26.3 

5.6 

10.0 

17.2 

11.1 

0.0 

0.0 

15.8 

10.5 

11.8 

5 

4.2 

0.0 

0.0 

0.0 

13.8 

5.6 

0.0 

4.8 

0.0 

5.3 

5.9 

6 

0.0 

0.0 

0.0 

0.0 

6.9 

0.0 

5.9 

0.0 

10.5 

0.0 

0.0 

7 

0.0 

0.0 

0.0 

0.0 

3.4 

0.0 

0.0 

0.0 

5.3 

0.0 

0.0 

8 

0.0 

5.3 

0.0 

0.0 

6.9 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

9 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

10 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

Median 

2 

3 

2 

2 

3 

2 

2 

1 

2 

2 

2 

Total 

24 

19 

18 

20 

29 

18 

17 

21 

19 

19 

17 
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4.3.1 VFR approach, all pilots 

For all the display concepts, the best rating received was a 1 “Excellent, Highly Desirable” 
characteristics. The PBG and SVS display concepts received a rating of 3 “Fair, Some Mildly Unpleasant 
Deficiencies” as the worst rating. The worst rating received for the BRD display concept was a 5 
“Moderately Objectionable Deficiencies”. One pilot rated the SCFD as having characteristics of an 8 
“Major Deficiencies”. For the VFR approach, DRR results indicate that all display concepts were rated as 
“satisfactory without improvement” or better (i.e., DRR ratings of 1 or a 2) at least 60% of the time as 
indicated in Figure 11. Display readability rating results for the SVS display concept indicate that it was 
rated as a 1 “Excellent, Highly Desirable” characteristics 45% of the time and either a 1 or a 2 “Good, 
Negligible Deficiencies” 80% of the time. The BRD and PBG display concept results needed to also 
include a rating of 3 “Fair, Some Mildly Unpleasant Deficiencies” to capture at least 80% of the ratings. 
The SCFD display concept results needed to also include ratings of 4 “Minor but Annoying 
Deficiencies” to capture 80% of the ratings. All display concepts, except the SCFD received a median 
DRR rating of 2 which is “good with negligible deficiencies” for this approach. The SCFD display 
concept received a median DRR rating of 3, which is “fair, some mildly unpleasant deficiencies” due to 
frustration involving the inability to differentiate lateral flight path from flight-path track-angle errors. 

4.3.2 ILS approach, all pilots 

For all the display concepts, the best rating received was a 1 - “Excellent, Highly Desirable” 
characteristics. The worst rating received for the SCFD and SVS display concepts was a 5 “Moderately 
Objectionable Deficiencies”. The worst rating for the PBG display concept was a 6 “Very Objectionable 
But Tolerable Deficiencies”. The BRD display concept received several ratings of 8 “Major 
Deficiencies” for the ILS approach as the worst rating. Display readability results indicate that all display 
concepts, except the BRD, were rated as “satisfactory without improvement” or better (i.e., DRR ratings 
of 1 or a 2) at least 60% of the time as indicated in Figure 11. The BRD display concept was rated as 
having “Minor But Annoying Deficiencies” or better (i.e., DRR ratings of 1 through 4) at least 60% of the 
time and having “Moderately Objectionable Deficiencies” (i.e., DRR ratings of 1 through 5) or better 
80% of the time. The SVS display concept was rated as a 1 having “Excellent, Highly Desirable” 
characteristics 57% of the time and DRR ratings of either a 1 or a 2 80% of the time. The PBG and 
SCFD display concepts were rated as “Fair, Some Mildly Unpleasant Deficiencies” or better 80% of the 
time. The SVS display concept was the only one to achieve 80% of the ratings being either a 1 or a 2. 

The SVS display concept was also the only one to receive a median rating of 1 for the ILS approach. 
Comparing the PBG and SVS display concepts reveals that the effect of the SVS terrain lead to a 40% 
increase in the percentage of DRR 1 ratings. It is believed that the presence of the SVS terrain facilitated 
integration of the other information elements on the display leading to the improved ratings. The BRD 
display concept received a median DRR rating of 3 “Fair, Some Mildly Unpleasant Deficiencies” for the 
ILS approach and the SCFD and PBG display concepts received median ratings of 2. 

4.3.3 VMC-Like approach, all pilots 

For all the display concepts, the best rating received was a 1 “Excellent, Highly Desirable” characteristics. 
The worst rating received for the PBG and SVS display concepts was a 5 “Moderately Objectionable 
Deficiencies”. The worst rating for the SCFD display concept was a 7 “Major Deficiencies”. All display 
concepts were rated as a 2 with “Good, Negligible Deficiencies” or better at least 60% of the time as 
shown in Figure 11. Only the SVS and PBG display concepts received ratings of a 3 “Fair, Some Mildly 
Unpleasant Deficiencies” or better 80% of the time. The SCFD display concept received ratings of a 4 
“Minor but Annoying Deficiencies” or better 80% of the time. It is believed that the lack of turn 
anticipation of the SCFD display concept contributed towards the degraded DRR results. Comparing the 
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PBG with the SVS display DRR results in Figure 1 1 concept revealed that the effect of SVS terrain 
increased the percentage of DRR 1 ratings by 17%. All display concepts received median ratings of 2 
“Good, Negligible Deficiencies”. 

4.3.4 ILS approach without low-time VFR pilot data 

The DRR data were also analyzed for the ILS approach without the VFR pilot data. Table 21 presents 
the percentage of the time pilots selected a specific DRR rating for each display concept for the ILS 
approach without the VFR pilot data (along with the median and total number of data points). The best 
rating received for all four display concepts was a 1 - “Excellent, Highly Desirable” characteristics. The 
worst ratings for the SVS and SCFD display concepts was a 5 “Moderately Objectionable Deficiencies” 
whereas the worst rating for the PBG display concept was a 6 “Very Objectionable But Tolerable 
Deficiencies”. As was illustrated for all the pilots’ data, the effect of SVS terrain contributed to increase 
the percentage of DRR 1 ratings by approximately 40% compared to the PBG display concept. This 
result is a similar result to the data for all of the pilots. However, the SVS display was one DRR 1 rating 
short of receiving a median rating of 1. At least 60% and 80% of the time, the SVS display concept was 
rated as a 2 or better. The PBG display concept required the inclusion of DRR 3 “Fair, Some Mildly 
Unpleasant Deficiencies” results to capture at least 60% and 80% percent of the responses. The SCFD 
was rated as a 2 or better 60% of the time and a 3 or better 80% of the time and considered similar to the 
PBG results. The BRD display concept received the worst ratings and was rated as a 4 “Minor But 
Annoying Deficiencies” or better 60% of the time and a 5 “Moderately Objectionable Deficiencies” or 
better 80% of the time. Results in Table 21 show that all display concepts, except for the BRD, received 
a median DRR rating of 2 “Good, Negligible Deficiencies” while the BRD received a median DRR rating 
of 3 “Fair, Some Mildly Unpleasant Deficiencies”. 

Table 21: Display Readability Results for the ILS Approach, IFR and H-IFR pilots 


DRR 

ILS Approach 

BRD 

SCFD 

PBG 

SVS 

1 

10.5 

16.7 

9.1 

46.7 

2 

21.1 

58.3 

45.5 

40.0 

3 

26.3 

8.3 

36.4 

6.7 

4 

10.5 

8.3 

0.0 

0.0 

5 

15.8 

8.3 

0.0 

6.7 

6 

5.3 

0.0 

9.1 

0.0 

7 

5.3 

0.0 

0.0 

0.0 

8 

0.0 

0.0 

0.0 

0.0 

9 

0.0 

0.0 

0.0 

0.0 

10 

0.0 

0.0 

0.0 

0.0 

Median 

3 

2 

2 

2 

Total 

19 

12 

11 

15 
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4.3.5 ILS and VMC-Like approach analysis, all pilots, advanced displays 

Comparing the results for the ILS approach with the results from the VMC-Like approach reveals that 
the best rating received by the SCFD, PBG, and SVS display concepts were identical. The worst rating 
received for the SCFD display concept increased to a 7 (“Major Deficiencies”) from a 5 (“Moderately 
Objectionable Deficiencies”) for the ILS approach. The worst rating received for the PBG display 
concept improved from a 6 (“Very Objectionable But Tolerable Deficiencies”) to a 5 (“Moderately 
Objectionable Deficiencies”) and the worst rating for the SVS display concept remained unchanged at a 
DRR of 5. Comparing the DRR results in Figure 1 1 for the ILS and VMC-Like approaches reveals that 
the SCFD, PBD, and SVS display concepts were rated as a 2 (“Good, Negligible Deficiencies”) at least 
60% of the time. There was a slight increase in the ratings for the SVS and SCFD display concepts for 
the VMC-Like approach compared to the ILS approach. The SVS display received a rating of 3 (“Fair, 
Some Mildly Unpleasant Deficiencies”) or better 80% of the time instead of a 2. The SCFD display 
concept was rated as a 4 “Minor But Annoying Deficiencies” or better 80% of the time instead of a 3. 

The SVS display concept median rating increased to a 2 from a 1 for the VMC-Like approach compared 
to the ILS approach. The median ratings for the SCFD and PBG display concepts was a 2 “Good, 
Negligible Deficiencies” for both the ILS and VMC-Like approaches. These increases in ratings are 
attributed to the higher workload of the VMC -Like approach. 

4.3.6 Test condition analysis 

Test condition analysis was conducting using a 4-level ranking method. First, the TCs were ranked 
based on median DRR received, then on the DRR required to capture at least 60% of the results, then on 
the DRR required to capture 80% of the results. Lastly, the TCs were ranked based on the maximum 
DRR rating received. The results of this analysis is presented in Table 22. The most favorably rated TCs 
were those for the VFR pilots only for the ILS and VMC-Like approaches with the SVS display concept 
that received median DRR results of 1 - “Excellent, Highly Desirable” characteristics. It is believed that 
the VFR pilots rated these conditions favorably due to the presence of the SVS terrain that provided them 
with flight information in an intuitive out-the-window manner they are most accustomed to. The ILS 
approach with the SVS display concept for all pilots also received a median DRR rating of 1. The VFR 
approach test conditions with the SVS and PBG display concepts received median DRR ratings of 2 
(“Good, Negligible Deficiencies”). The presence of the SVS terrain on the SVS display concept is 
believed to provide a higher level of information integration during IMC conditions than in VMC 
conditions which leads to the lower DRR ratings for most of the IMC TCs. The SCFD was rated slightly 
better than the PBG display concept for the ILS approach but slightly worse for the VMC-Like approach. 
It is believed that the presence of the highway-in-the-sky tunnel of the PBG display concept provides 
substantial benefits for curved approaches. The presence of the SVS terrain provided a larger reduction in 
the DRR ratings for the ILS approach than for the VMC-Like approach. For the ILS approach the 
reduction of DRR ratings due to the SVS terrain resulted in a lowering of the median rating to a 1 from a 
2 as well as reducing the range of DRR rating required to capture 80% results and maximum ratings. For 
the VMC-Like approach PBG and SVS displays concepts were rated identically to each other. All of the 
IMC TCs were rated better than the ILS + BRD display concept baseline, which does not include VFR 
pilots data for the test condition analysis. This indicates that the SCFD, PBG, and SVS display concepts 
provide improved display readability for the ILS approach, compared to what is currently the minimum 
required equipage, and could be expected to facilitate advanced IMC approaches similar to the VMC-Like 
approach considered herein. 
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Table 22: DRR Results for Test Condition. 


Test Condition Name 

TC 

Median 

<60% 

<80% 

Max 

ILS Approach + SVS (only VFR pilots) 

12 

1 

1 

1 

3 

VMC-Like Approach + SVS (only VFR pilots) 

13 

1 

1 

3 

4 

IFS Approach + SVS 

8 

1 

2 

2 

5 

VFR Approach + SVS 

4 

2 

2 

2 

4 

VFR Approach + PBG 

3 

2 

2 

3 

4 

IFS Approach + SCFD 

6 

2 

2 

3 

5 

VMC-Fike Approach + PBG 

10 

2 

2 

3 

5 

VMC-Fike Approach + SVS 

11 

2 

2 

3 

5 

IFS Approach + PBG 

7 

2 

2 

3 

6 

VMC-Fike Approach + SCFD 

9 

2 

2 

4 

7 

VFR Approach + BRD 

1 

2 

3 

3 

5 

VFR Approach + SCFD 

2 

3 

3 

4 

8 

IFS Approach + BRD 

5 

3 

3 

5 

7 


4.4 Terrain Awareness 

The EPs were asked to rate their overall understanding of the terrain environment within which they 
were operating. Items to consider were the quantity and quality of the terrain information available to 
them and how comfortable they were with their terrain awareness. These data were acquired to augment 
other results and facilitate the determination of the effect that terrain awareness provides to overall SA. 
Terrain Awareness (TA) ratings for the four display concepts and three approaches are presented in 
Figure 12 and Table 23 and were generated through use of the TA question in Appendix C. (Error bars in 
Figure 12 represent one SD.) Individual TA results for each evaluation are provided in Appendix D. 
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Table 23: TA Descriptive Statistics (all Pilots) 


Display 

Concept 

Approach 

Mean 

Standard 

Deviation 

Number of Data 
Points 

BRD 

VFR 

84.5 

11.9 

24 


1LS 

9.2 

9.7 

29 


Total 

48.0 

40.2 

37 

SCFD 

VFR 

61.3 

26.2 

19 


1LS 

48.1 

22.5 

18 


VMC-Like 

46.1 

23.0 

18 


Total 

52.0 

24.5 

55 

PBG 

VFR 

69.0 

23.5 

18 


ILS 

51.7 

30.2 

17 


VMC-Like 

55.4 

28.7 

19 


Total 

58.8 

28.1 

54 

svs 

VFR 

87.8 

10.6 

19 


ILS 

84.8 

17.5 

21 


VMC-Like 

79.7 

25.1 

19 


Total 

84.1 

18.6 

59 

Total 

VFR 

75.9 

22.0 

75 


ILS 

49.8 

34.5 

74 


VMC-Like 

60.7 

29.1 

56 


Total 

60.2 

32.1 

221 
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Figure 12: Terrain Awareness Results for All Display Concepts, All Pilots and All Approaches 


4.4.1 VFR approach discussion 

For the VFR approach, TA ratings were high for the BRD display concept (84.5) but decreased for the 
SCFD and PBG display concepts. Even though evaluation pilots were instructed to fly a VFR approach 
using the out-the-window environment for navigation, EPs occasionally used the guidance provided by 
the advanced display concepts (SCFD, PBG, and SVS) to cross check their flight path. An ANOVA 
conducted on the TA results for the VFR approach for the main effects of display concept and pilot type 
resulted in highly significant results for display concept (F(3,80)=8.75, p<0.01) but not for pilot type. A 
subsequent SNK post-hoc analysis resulted in two unique subsets shown in Table 24. Evaluation pilots 
noted that the increased attention allocated to looking at the displays for cross-check information versus 
out-the-window navigation reduced their terrain awareness, especially for the SCFD display concept. In 
Table 24 terrain awareness significantly decreased 15.5 units for the PBG display concept compared to 
the BRD. Terrain awareness also decreased another 7.7 units for the SCFD compared to the PBG display 
concept. However, for the SVS display concept, terrain awareness ratings were slightly higher than for 
the BRD display concept. Traditionally pilots needed to look outside the aircraft to see terrain, and other 
threats such as traffic, and inside the aircraft for flight and navigation information. The ratio of the time 
pilots look outside the aircraft to within is frequently used as a evaluation criterion for new avionics and 
procedures. However, the presence of SVS terrain on the PFD alters this paradigm since terrain 
information is provided in both eyes-out and eyes-in locations. 
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Table 24: Mean TA SNK Results for Display Concept, for all Pilots for the VFR Approach 


Display 

Concept 

Number of 
Data Points 

Subset 

1 

2 

SCFD 

19 

61.3 


PBG 

18 

69.0 


BRD 

24 


84.5 

SVS 

19 


87.8 

Significance 


0.20 

0.58 


4.4.2 ILS approach, all pilots 

An ANOVA conducted on the TA results for the ILS approach for the main effects of display concept 
and pilot type resulted in highly significant results for display concept (F(3,85)=59.18, p<0.01) but not for 
pilot type. A subsequent SNK post-hoc analysis resulted in three unique subsets shown in Table 25. 

From Table 25 it can be seen that the BRD display concept was rated much worse in terms of TA with a 
38.9 unit decrease compared to the SCFD display concept for the ILS approach and was the only member 
of this subset. This reflects the EPs very low awareness of the terrain environment within which they 
were operating with the BRD display concept without out-the-window visibility. For this condition, EPs 
created a mental model of the terrain outside the aircraft through their knowledge of where they were 
along the ILS approach path combined with their mental recall of the operational area and paper charts. 
Compared to the BRD display concept, the SCFD and PBG display concepts yielded significant increases 
in terrain awareness of 38.9 and 42.5 units respectively, and were the only two members of the 
intermediate terrain awareness subset, reflecting the incorporation of the ND. Terrain awareness for the 
SVS display concept was significantly increased by 33.1 units compared to the PBG display concept for 
the ILS approach. This result is due to the presence of a perspective display of 3-D terrain on the PFD 
and its intuitive integration with the flight information and guidance symbology. Note that the level of 
terrain awareness for the SVS display concept for the ILS approach was virtually identical to the level of 
terrain awareness for the BRD display concept for the VFR approach as shown in Figure 12. 

Table 25: Mean TA SNK Results for Display Concept, for all Pilots for the ILS Approach 


Display 

Concept 

Number of 
Data Points 

Subset 

1 

2 

3 

BRD 

29 

9.2 



SCFD 

18 


48.1 


PBG 

17 


51.7 


SVS 

21 



84.8 

Significance 


1.00 

0.55 

1.00 
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4.4.3 VMC-Like approach, all pilots 

An ANOVA conducted on the TA results for all pilots and display concepts revealed that the main 
effect was highly significant for display concept (F(2,56)=10.1, p<0.01) but not for pilot type (p>0.05). 
There were no significant interactions between the main effects. An SNK post-hoc analysis on the TA 
results provided two unique subsets for display concept as shown in Table 26. Even though the outside 
terrain was provided in a gods-eye-view format on the ND, the addition of the SVS terrain on the PFD 
significantly improved perceived terrain awareness for the SVS display concept compared to the PBG 
display concept. Overall, the SVS display concept was the only member of the highest (best) terrain 
awareness subset due to the effective integration of terrain with guidance and flight information 
symbology. The improvement in terrain awareness due to guidance symbology for the PBG display 
concept was approximately 9 units and not statistically different compared to the SCFD display concept. 
It is likely that the improved terrain awareness was related to the reduced workload for the PBG display 
concept that allowed pilots more time to absorb terrain information from the ND. 

Table 26: Mean TA SNK Results for Display Concept for all Pilots for the VMC-Like Approach 


Display 

Concept 

Number of 
Data Points 

Subset 

1 

2 

SCFD 

18 

46.1 


PBG 

19 

55.4 


SVS 

19 


79.7 

Significance 


0.228 

1.00 


4.4.4 ILS approach without low-time VFR pilot data 

Results for TA were analyzed without the low-time VFR pilot data, enabling a more direct comparison 
of the four display concepts within current operations. Results from an ANOVA indicated that display 
concept was highly significant (F(3,56)=43.26, p<0.01) for TA. Pilot type was not significant for TA 
(p>0.05). As can be seen in the Table 27, three unique subsets were formed. The lowest (worst) TA was 
for the BRD display concept and represents the nominal level of terrain awareness for operations with 
traditional avionics. Compared to the BRD display concept, the incorporation of the gods-eye view of 
terrain on the ND combined with more intuitive flight path information significantly increased terrain 
awareness 39.9 and 47.7 units for the SCFD and PBG display concepts, respectively. However, for the 
SVS display concept, terrain awareness increased an additional 26.8 units compared to the PBG display 
concept. Subsequent SNK results, provided in Table 27, show that the SVS display concept was the only 
member of the highest (best) TA rating group. The SCFD and PBG display concepts were statistically 
similar and were in the middle subset. 
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Table 27: Mean TA SNK Results for the ILS Approach, IFR and H-IFR Pilots 


Display 

Concept 

Number of 
Data Points 

Subset 

1 

2 

3 

BRD 

18 

10.9 



SCFD 

12 


50.8 


PBG 

11 


58.6 


SVS 

15 



85.5 

Significance 


1.00 

0.30 

1.00 


4.4.5 ILS and VMC-Like approach analysis, all pilots, advanced displays 

In an effort to compare the advanced display concepts with each other and also to evaluate the 
potential of these displays to support an advanced IMC approach for all pilot types, an ANOVA was 
performed on the TA data for the advanced displays with all pilots’ data for the ILS and VMC-like 
approaches. Results indicated that display concept was highly significant (F(2,l 12)=29.606, p<0.01) for 
this measure. Pilot type was significant (F(2,l 12)=2.91, p=0.01), but approach was not for the TA 
ratings. Post-hoc SNK analysis revealed no significant TA differences for pilot type. However, in Table 
28, it can be seen that the VFR pilots reported lower TA than the IFR pilots, and that the IFR pilots 
reported lower TA than the FI-IFR pilots. This trend could be attributed to pilots’ experience, with the 
more experienced pilots having the ability to better control the aircraft while absorbing terrain 
information from the various displays. 

Table 28: Mean TA SNK Results for Pilot Type for the ILS and VMC-Like Approach for all Pilots 


EP-type 

Number of 
Data Points 

Mean 

VFR 

36 

57.7 

IFR 

37 

61.6 

H-IFR 

39 

65.8 

Significance 


0.22 


Post-hoc SNK analysis for display concept revealed two unique subsets for the TA ratings (see Table 
29). The SVS display concept was rated as providing the most terrain awareness. While the mean TA 
ratings were higher for the PBG display concept compared to the SCFD concept, the difference was not 
enough to be significant. The addition of SVS terrain appears to have increased TA for the two IMC 
approaches, with an increase in TA of almost 29 units. 
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Table 29: Mean TA SNK Results for Display Concept for the ILS and VMC-Like Approach, All 

Pilots 


Display 

Concept 

Number of 
Data Points 

Subset 

1 

2 

SCFD 

36 

47.1 


PBG 

36 

53.7 


SVS 

40 


82.4 

Significance 


0.18 

1.00 


4.4.6 Test condition analysis 

A subsequent analysis on TA was performed based on TC with post-hoc results presented in Table 30. 
Test Condition was highly significant for TA results (F(12,221)=20.583, p<0.01). Four overlapping 
subsets were formed from post-hoc analysis. The highest (best) rated group included all of the VFR test 
conditions, as would be expected, except for the SCFD display concept. As noted previously, evaluation 
pilots reported some frustration with the SCFD for the VFR approach. As a result TC 2 (VFR approach 
with SCFD) had lower TA results, which put it in the next lower subset. All of the simulated 1MC 
evaluations of the SVS display concept were grouped with the highly rated VFR evaluations. This 
indicates that, for TA, out- the- window visibility had little effect for the SVS display concept. Terrain 
Awareness results may also indicate that advanced 1MC approaches, like the VMC-like approach, are 
feasible with SVS. 

The worst TA subset was comprised of only TC 5 (ILS approach evaluation of the BRD display 
concept). While this type of result has been observed for the other variables (i.e., SART, TLX, DRR), the 
separation of TC 5 from the others is much more pronounced for TA. Subset 2 is comprised of ILS and 
VMC-like evaluations of the SCFD and PBG display concepts and is only slightly different in 
composition than subset three. Note that subset three includes TC 3 (VFR approach with the PBG display 
concept) and doesn’t include TC 9 (VMC-like approach with the SCFD display concept). Overall, the TC 
analysis of terrain awareness indicates that all the advanced display concepts provided superior TA than 
TC 5 (ILS approach, BRD display concept) for both ILS and VMC-like approaches, most likely due to 
the presence of terrain on the ND. However, the SVS display concept provides additional significant 
improvements in terrain awareness with results for simulated IMC that are as good or better than those for 
the VFR approach with the BRD. 
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Table 30: Mean TA SNK Results for Test Condition 


Test Condition Name 

TC 

# 

Number 
of Data 
Points 

Subset 

1 

2 

3 

4 

ILS Approach + BRD (no VFR pilots) 

5 

18 

10.9 




VMC-Like Approach + SCFD 

9 

18 


46.1 



ILS Approach + SCFD 

6 

18 


48.1 



ILS Approach + PBG 

7 

17 


51.7 

51.7 


VMC-Like Approach + PBG 

10 

19 


55.4 

55.4 


VFR Approach + SCFD 

2 

19 


61.3 

61.3 


VFR Approach + PBG 

3 

18 



69.0 

69.0 

VMC-Like Approach + SVS 

11 

19 




79.7 

VMC-Like Approach + SVS (only VFR pilots) 

13 

6 




80.7 

ILS Approach + SVS (only VFR pilots) 

12 

6 




82.8 

VFR Approach + BRD 

1 

24 




84.5 

ILS Approach + SVS 

8 

21 




84.8 

VFR Approach + SVS 

4 

19 




87.8 

Significance 



1.0 

0.2 

0.1 

0.1 
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4.5 PTS Results 


This section discusses the results against IFR Practical Test Standard (PTS) criteria. Recall that the 
PTS criteria are composed of a combination of lateral, vertical, and airspeed errors. PTS results are 
presented in terms of a percentage of time the evaluation pilots were able to satisfy all three criterion 
simultaneously during Segments 2 and 3 (i.e., from points A and B in Figure 8 down to 200 ft AGL). 

PTS results are shown in Figure 13 and presented in Table 31. (In Figure 13, the error bars represent one 
SD.) Subsequent sections of this report discuss each of the parameters employed for the PTS criteria 
analyses, sometimes focusing only on the final approach segment of the total approach (i.e., Segment 3, 
simultaneous localizer and glideslope tracking). 

Table 31: PTS Descriptive Statistics (% Time), Segments 2 and 3 for all Pilots 


Display 

Concept 

Approach 

Mean 

Standard 

Deviation 

Number of 
Data Points 

BRD 

1LS 

73.4 

23.7 

30 


Total 

73.4 

23.7 

30 

SCFD 

1LS 

88.2 

9.9 

18 


VMC-Like 

89.5 

10.5 

18 


Total 

88.8 

10.1 

36 

PBG 

1LS 

90.4 

9.8 

16 


VMC-Like 

90.1 

12.5 

19 


Total 

90.2 

11.2 

35 

svs 

1LS 

90.7 

7.9 

27 


VMC-Like 

92.0 

10.5 

19 


Total 

91.2 

9.0 

46 

Total 

ILS 

84.5 

17.2 

91 


VMC-Like 

90.5 

11.1 

56 


Total 

86.8 

15.4 

147 
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Figure 13: Percent Time within IFR PTS Performance Metrics, All Pilots, Segments 2 and 3 


4.5.1 ILS approach, all pilots 

An ANOVA conducted on the PTS data for all pilots for the ILS approach for the main effects of 
display concept and pilot type resulted in highly significant results for display concept (F(3,85)=8.28, 
p<0.01) and significant results for pilot type (F(2,85)=3.64, p=0.03). There were no significant 
interactions between the main effects. Subsequent SNK post-hoc analysis revealed two unique subsets 
for both display concept and pilot type as shown in Tables 32 and 33. The BRD display concept was the 
only member of the worst PTS performance group. The incoiporation of the ND, with desired flight path 
and ownship symbology, combined with the SCFD guidance or FIITS pathway presentations of the 
advanced display concepts, greatly facilitated the evaluation pilots’ ability to remain within the PTS 
criteria. All of the advanced display concepts (i.e., SCFD, PBG, and SVS) were included in the best PTS 
performance group. A significant 14.8% improvement in PTS results occurred for the SCFD display 
concept compared to the BRD. The PBG and SVS display concepts improved PTS results by 2.2% and 
2.9%, respectively, compared to the SCFD. 

As anticipated for the ILS approach, the VFR pilots’ PTS performance was significantly worse than 
the IFR and H-IFR pilots. Table 34 presents PTS results for display concept and pilot type and the 
difference between results for the VFR pilots compared to the average results for the IFR and H-IFR pilot 
groups and shows that differences between the different pilot types were much larger for the BRD display 
concept than the advanced display concepts. The decrease in VFR pilot performance was 4.6 times worse 
(i.e., 23.7/5.2=4.6) for the BRD display concept compared to SCFD and 9.9 times (i.e., 23.7/2.4=9.9) 
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worse than the SVS display concept. This result can be attributed to the VFR pilots’ lack of training in 
integrating disparate information from the individual BRD display gauges. However, the highly- 
integrated and intuitive display of information of the advanced display concepts greatly improved the 
VFR pilots ability to fly without outside visual cues. This was especially true for the SVS display 
concept. Subsequent sections of this report provide analyses of the constituent elements of the PTS 
criteria (i.e., localizer, glideslope and IAS errors) providing insight into the distribution of exceedances 
along the approach. 

Table 32: Mean PTS (% of Time) SNK Results for Display Concept for all Pilots, ILS Approach. 


Display 

Concept 

Number of 
Data Points 

Subset 

1 

2 

BRD 

30 

73.4 


SCFD 

18 


88.2 

PBG 

16 


90.4 

SVS 

21 


91.1 

Significance 


1.00 

0.797 


Table 33: Mean PTS (% of Time) SNK Results for Pilot Type for All Display Concepts, ILS 

Approach, 


Pilot Type 

Number of 
Data Points 

Subset 

1 

2 

VFR 

29 

75.9 


H-IFR 

27 


88.0 

IFR 

29 


88.7 

Significance 


1.00 

0.849 
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Table 34: Mean PTS (% of Time) Results for Pilot Type and Display Concept, ILS Approach. 


Display 

Concept 

RH 

■MS 

Mean 

VFR pilots results compared to the 
average of IFR and H-IFR pilots 

Number of Data 
Points 

BRD 

VFR 

58.5 

-23.7 

11 

IFR 

82.8 


11 

Fl-IFR 

81.1 


8 

SCFD 

VFR 

84.7 

-5.2 

6 

IFR 

90.8 


6 

Fl-IFR 

89.0 


6 

PBG 

VFR 

85.7 

-6.7 

6 

IFR 

94.4 


4 

Fl-IFR 

92.5 


6 

SVS 

VFR 

89.4 

-2.39 

6 

IFR 

92.4 


8 

Fl-IFR 

91.1 


7 


4.5.2 VMC-Like approach, all pilots, advanced displays 

An ANOVA conducted on the PTS results for the main effects of display concept (F(2,56)=1.25, 
p=0.30) and pilot type (F(2,56)=0.88, p=0.42) resulted in no significant (p<0.05) differences for either 
and no significant interactions. Tables 35 and 36 provide the mean results for display concept and pilot 
type. While not statistically significant, the SVS display concept enabled pilots to remain within the PTS 
criteria 2.5% longer than the SCFD. In addition, the VFR pilots were able to generate similar PTS results 
to the IFR pilots and only 3.9% worse than the Fl-IFR pilots, even though the VMC-like approach 
included two 90-degree descending turns during the last 3.4 nm of the approach. 

Table 35: Mean PTS (% of Time) SNK Results for Display Concept for all Pilots, VMC-Like 

Approach 


Display Concept 

Number of 
Data Points 

Subset 

SCFD 

18 

89.5 

PBG 

19 

90.1 

SVS 

19 

92.0 

Significance 


0.503 
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Table 36: Mean PTS (% of Time) SNK Results for Pilot Type for All Display Concepts, VMC-Like 

Approach 


Pilot Type 

Number of 
Data Points 

Subset 

IFR 

18 

89.0 

VFR 

18 

89.2 

H-IFR 

20 

93.1 

Significance 


0.169 


4.5.3 ILS approach Segment 3 without VFR pilot data 

An ANOVA was conducted on the ILS approach Segment 3 results without the VFR pilot data for the 
main effects of display concept and pilot type. Results of the ANOVA indicated no significant effect of 
display concept or pilot type (p>0.05). It is likely that instrument training for the IFR and Fl-IFR pilots 
enabled them to maintain acceptable performance even with the BRD display concept. Flowever, while 
the results for the display concept were not statistically significant, the BRD display concept tended 
toward the worst PTS performance of the tested concepts, as shown in Table 37. IFR and Fl-IFR pilots 
were only able to satisfy all three of the PTS criteria simultaneously 83.1% of the time. This establishes 
the baseline minimum required PTS performance within the data set considered herein. The SCFD 
display concept improved PTS results by 7.8% compared to the BRD and reflects the improved 
capabilities due to the ND and integrated PFD with the single-cue flight director. The PBG and SVS 
display concepts improved PTS results by 1.8% and 3.3% compared to the SCFD display concept, 
respectively. 

Table 37: PTS (% of Time) Results for Display Concept for Segment 3 of the ILS Approach, IFR 

and H-IFR Pilots Only 


Display 

Concept 

Mean 

BRD 

83.1* 

SCFD 

89.2 

SVS 

90.9 

PBG 

92.0 


*Note: This value is used herein as the baseline performance for an ILS Approach. 

In contrast to the analysis with the VFR pilot data, the effect of display concept was substantially 
reduced, as can be seen through comparison of the ANOVA results as well as Tables 32 and 37. This 
result indicates that the VFR pilots had a much harder time with the BRD display concept than the 
instrument rated pilots. 

4.5.4 ILS and VMC-Like approaches, Segment 3, all pilots, advanced displays 

Figure 14 presents PTS criteria results for Segment 3 of the ILS- and VMC-Like approach maneuvers 
for all display concepts. An ANOVA was performed on the PTS data for the advanced displays for 
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segment 3 of the ILS and VMC-like approaches in an effort to compare the advanced display concepts 
with each other and also to evaluate their ability to support advanced IMC approaches for all pilot types. 
Display concept, approach, and pilot type were not significant (p>0.05) for this analysis and there were no 
significant interactions. While not significant, Tables 38, 39, and 40 provide information to assess the 
interactions between display concept, approach and pilot type. General trends in the data indicate that the 
VFR pilots performed 2 to 6% worse than the IFR and Fl-IFR pilots and that the effect of pilot type was 
the smallest for the SVS display concept. Overall, PTS performance results for the VMC-like approach 
with the advanced display concepts were as good or better than the acceptable PTS performance for the 
ILS approach as established herein. This result indicates that the advanced display concepts can 
potentially enable pilots to accurately perform approaches other than classic ILS-type approaches, with 
long straight legs, across a broad range of pilot skills. 



Approach 


Figure 14: Percent Time within IFR PTS performance Metric, All Pilots, Segment 3 Only 
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Table 38: PTS (% of Time) Criteria for Display Concept and Approach interactions, ILS and 

VMC-Like Approaches, Segment 3 


Display 

Concept 

Approach 

Mean 

SCFD 

ILS 

87.8 

VMC-Like 

85.4 

PBG 

ILS 

89.6 

VMC-Like 

85.5 

SVS 

ILS 

89.9 

VMC-Like 

92.1 


Table 39: PTS (% of Time) Criteria Results for Display Concept and Pilot Type Interactions, ILS 

and VMC-Like Approaches, Segment 3 


Display 

Concept 

Pilot Type 

Mean 

SCFD 

VFR 

84.3 

IFR 

86.8 

H-IFR 

88.7 

PBG 

VFR 

84.5 

IFR 

86.8 

H-IFR 

90.4 

SVS 

VFR 

88.9 

IFR 

91.3 

H-IFR 

92.4 


Table 40: PTS (% of Time) Criteria Results for Approach and Pilot Type Interactions, ILS and 

VMC-Like Approaches, Segment 3 


Approach 

Pilot Type 

Mean 

ILS 

VFR 

86.1 


IFR 

91.7 


H-IFR 

89.6 

VMC-Like 

VFR 

85.7 


IFR 

85.4 


H-IFR 

91.5 
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4.5.5 Test condition analysis, Segment 3 

An ANOVA was performed on the PTS results for TC with SNK post-hoc results presented in Table 
40. The analysis was limited to segment 3. Test Condition was not significant (p>0.5) for PTS results. 
From Table 40, it can be seen that the worst PTS results were for TC 5 (ILS approach with BRD display 
concept) and the best results were for TC 1 1 (VMC-lIke approach for the SVS display concept). Of the 
simulated-IMC evaluations, TC 5 was again the worst while all combinations of advanced display 
concepts and IMC approaches (i.e., ILS or VMC-like) were better, though not significantly. Not only did 
the advanced display concepts slightly improve PTS results compared to the BRD, but this improvement 
was accomplished when also including the VFR pilot data. Note that TC 8 (ILS approach with SVS 
display concept) was 2.2% worse than TC 1 1 (VMC-like approach with SVS display concept). One 
possible explanation for this result involves the VFR pilots and their increased familiarity with the VMC- 
like approach, due to its similarity to the VFR approach, compared to the ILS approach. Occasionally, 
VFR evaluation pilots would forget to reduce their airspeed from 100 kts to 90 kts when intercepting the 
glideslope for the ILS approach, but never missed the airspeed reduction at waypoint NEWT for the 
VMC-like approach. Recall that waypoint NEWT was located abeam the runway numbers on the 
downwind leg and was a significant event location for the VFR approach. It is also worthwhile to note 
that the effect of the SVS terrain led to a 7% improvement in evaluation pilots’ performance compared to 
the PBG display concept for the VMC-like approach. This is likely due to the increased awareness 
provided by the SVS terrain background for this highly dynamic approach segment. It is believed that the 
SVS terrain served as an integrator for the other disparate pieces of information provided on the PFD (i.e., 
attitude, altitude, flight path, heading, etc.) and was particularly valuable for the VFR pilots. Further 
discussion regarding the elements of the PTS metric are provided in subsequent sections. In particular it 
is shown that the effect of the SVS terrain enables a significant improvement in the pilots ability to 
minimize glideslope error. Results for the VFR pilot evaluations of the SVS display concept for the ILS 
and VMC-like approaches were better than TC 5 (ILS approach with BRD display concept). Although 
the PTS results were not statistically significant, they indicate that advanced IMC approaches may be 
possible for the advanced display concepts, especially for the SVS display concept. 

Table 41: Mean PTS (% of Time) SNK Criteria Results for TC, All pilots, Segment 3 


Test Condition Name 

TC 

Number of 
Data Points 

Mean 

ILS Approach + BRD (no VFR pilots) 

5 

19 

83.1 

VMC-Like Approach + SCFD 

9 

18 

85.4 

VMC-Like Approach + PBG 

10 

19 

85.5 

ILS Approach + SVS (VFR pilots) 

12 

6 

87.7 

ILS Approach + SCFD 

6 

18 

87.8 

ILS Approach + PBG 

7 

16 

89.6 

ILS Approach + SVS 

8 

21 

89.9 

VMC-Like Approach + SVS (VFR pilots) 

13 

6 

90.2 

VMC-Like Approach + SVS 

11 

19 

92.1 

Significance 



0.715 
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4.6 Localizer Error 


Using localizer error results to describe the differences between the different display concepts and 
approaches is beneficial because the large majority of traditional IMC operations employ this type of 
guidance strategy. It is an especially valuable parameter to work with since the BRD display concept 
only displayed angular errors to the pilots. Comparing display concepts only on linear lateral errors 
substantially skews results towards the advanced display concepts since their guidance cue displacements 
are based on linear lateral error and are not affected by range to touchdown. Localizer error results for the 
four display concepts and three approaches are presented in Figure 15, 16, 17, and 18 along with Table 
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Table 42: Localizer Error (dots) Descriptive Statistics for all Pilots 


Display 

Concept 

Approach 

Mean 

Standard 

Deviation 

Number of 
Data Points 

BRD 

VFR 

2.78 

1.30 

19 


1LS 

0.75 

0.46 

18 


Total 

1.65 

1.37 

37 

SCFD 

VFR 

0.40 

0.37 

19 


1LS 

0.17 

0.09 

18 


VMC-Like 

0.43 

0.30 

18 


Total 

0.34 

0.30 

55 

PBG 

VFR 

0.18 

0.08 

18 


ILS 

0.09 

0.03 

16 


VMC-Like 

0.19 

0.07 

19 


Total 

0.16 

0.08 

53 

svs 

VFR 

0.20 

0.15 

18 


ILS 

0.10 

0.03 

20 


VMC-Like 

0.17 

0.07 

19 


Total 

0.15 

0.10 

58 

Total 

VFR 

1.03 

1.38 

79 


ILS 

0.34 

0.41 

85 


VMC-Like 

0.26 

0.22 

56 


Total 

0.57 

0.93 

220 


4.6.1 VFR approach, all pilots 

As can be seen in Figure 15 and Table 43, the VFR approach evaluation of the BRD display concept 
resulted in large localizer error values compared to the advanced display concepts. This increase in 
localizer error is again due to the EPs’ flight path navigation being driven totally by the out- the- window 
view for the BRD display concept. A primary source of localizer error for the VFR approach evaluation 
of the BRD display concept occurred during the base leg. EPs would fly base legs that sometimes were 
inside of the intended flight path while still maintaining reasonable VFR approach conditions. Providing 
EPs with moving maps, desired flight path and ownship representation as included in the navigation 
display, along with flight path error and flight path guidance symbologies on the advanced display 
concepts, greatly reduced localizer error, as indicated in Figure 15 and Table 43. This result was 
generated even though the EPs were primarily flying a VFR eyes-out approach and using the guidance 
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information to cross check their out-the-window navigation. Note that the PBG and SVS display 
concepts produced the lowest level of localizer error (approximately 0.19 dots mean). An ANOVA 
conducted on the localizer error data for the VFR approach for the main effects of display concept and 
pilot type resulted in highly significant results for display concept (F(3,79)=162.79, p<0.01) and non- 
significant results for pilot type (p>0.05). 

Table 43: Mean Localizer Error (dots) SNK results for Display Concept VFR Approach for All 

pilots 


Display 

Concept 

Number of 
Data Points 

Subset 

1 

2 

PBG 

18 

0.18 


SVS 

18 

0.20 


SCFD 

19 

0.40 


BRD 

24 


2.78 

Significance 


0.328 

1.000 
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Figure 15: VFR Approach, Localizer Error, By Display Concept and Pilot Group 
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4. 6.2 ILS approach, all pilots 

For the ILS approach for all pilots, the localizer error decreased for all display concepts compared to 
the results for the VFR approach, as shown in Figures 15 and 16 and Table 44. The largest decrease was 
for the BRD display concept, which decreased from 2.8 dots for the VFR approach to 0.49 dots for the 
ILS approach. This reduction was due primarily to the increased level of flight path precision associated 
with IMC operations where pilots relied entirely on flight path guidance and error cues to minimize flight 
path error, along with the different nature of the ILS approach with long, straight approach legs. Figure 
16 also shows that the effect of pilot group was eliminated for the advanced display concepts. This result 
was not the case for the BRD display concept, where the VFR pilot data indicates a difference compared 
to the IFR and H-IFR pilot data. Figure 1 6 also shows that the 1 .5 dot IFR PTS criterion was met for the 
advanced display concepts for both Segment 2 and 3 of the ILS approach and not met for the BRD 
display concept for the VFR pilots for Segment 3. Again, Segment 2 did not include specific guidance 
cues for the BRD display concept and pilots were merely flying a localizer intercept heading after 
approach initialization, then tracking the localizer and maintaining altitude until intercepting the 
glideslope. 

An ANOVA conducted on the localizer error data for the ILS approach for the main effects of 
display concept and pilot type revealed that display concept was highly significant (F(3,85)=40.9, p<0.01) 
and pilot type was not significant (p>0.05). There was a significant interaction between display concept 
and pilot type (F(6,85)=2.63, p=0.025). Post-hoc analysis resulted in the two subsets shown in Table 44. 
The worst subset was comprised of only the BRD display concept. The advanced display concepts 
comprised the best subset with the SCFD reducing localizer error by 77% compared to the BRD. The 
PBG and SVS display concepts reduced localizer error further compared to the SCFD by 41% and 47%, 
respectively. 

The relatively poor performance of the BRD compared to the advanced display concepts can be 
partially attributed to the lack of guidance during Segment 2 of the ILS approach, along with the 
incorporation of the VFR pilot data. Table 45 provides results to support the pilot/display-concept 
interaction and shows that for the BRD display concept, the VFR EPs flew much worse than the IFR and 
H-IFR pilots. The data seems indicative of the lack of IFR training for the VFR pilots. They did receive 
the same training associated with this flight test, but it can also be stated that for the VFR-rated EPs, the 
out-the-window view is key to integrating and assimilating all the information they are being presented. 
When that out-the-window view is no longer available, VFR EPs had a challenging time integrating the 
information presented via the BRD display concept. However, the integrated presentation of flight 
information on the advanced display concepts enabled the VFR pilots to produce localizer error results 
similar to the IFR and H-IFR pilot groups. 

Table 44: Mean Localizer Error (dots) SNK results for Display Concept for ILS Approach for all 

Pilots 


Display Concept 

Number of 
Data Points 

Subset 

1 

2 

PBG 

16 

0.09 


SVS 

21 

0.10 


SCFD 

18 

0.17 


BRD 

30 


0.75 

Significance 


0.596 

1.000 
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Figure 16: ILS Approach, Localizer Error, By Display Concept and Pilot Group 


69 


Table 45: Localizer Error Results (dots) for Display Concept and Pilot Type Interactions for the 

ILS Approach 


Display 

Concept 

Pilot Type 

Mean 

Standard 

Deviation 

Number of Data 
Points 

BRD 

VFR 

1.04 

0.51 

11 


IFR 

0.57 

0.28 

11 


H-IFR 

0.60 

0.41 

8 

SCFD 

VFR 

0.21 

0.08 

6 


IFR 

0.18 

0.11 

6 


H-IFR 

0.11 

0.08 

6 

PBG 

VFR 

0.10 

0.03 

6 


IFR 

0.08 

0.03 

4 


H-IFR 

0.10 

0.04 

6 

SVS 

VFR 

0.10 

0.05 

6 


IFR 

0.09 

0.03 

8 


H-IFR 

0.10 

0.03 

7 


4.6.3 VMC-Like approach, all pilots 

Figure 17 shows very different results for the three advanced display concepts for the VMC-like 
approach. While results are very similar for Segment 2 (i.e., the straight, constant-altitude leg), 
substantial differences are evident in Segment 3 (i.e., two descending 90-degree turns). The differences 
apparent for the SCFD display concept compared to the PBG and SVS display concepts are located at the 
turns to base leg and final approach and can be attributed to the lack of turn anticipation for this display 
concept. All of the advanced display concepts included the ND with the moving map, flight path and 
ownship symbology. It is evident from these data that the evaluation pilots were not looking at the ND 
very much during this approach, especially for Segment 3, due to the rapid pace and relatively high 
workload. An ANOVA conducted on the localizer data for the VMC-like approach for the main effects 
of display concept and pilot type revealed highly significant results for display concept (F(2,56)=13.69, 
p<0.01) and non-significant results for pilot type (p>0.05). There was no significant interaction between 
display concept and pilot type. Subsequent post-hoc analysis, as shown in Table 46, reveals two unique 
subset-groups. The lowest (best) subset was comprised of the SVS and PBG display concepts, while the 
SCFD was the only member of the higher (worst) subset with localizer error 2.4 times larger than the 
mean for the SVS and PBG display concepts. 

Table 46: Mean Localizer Error (dots) SNK results for Display Concept for the VMC-Like 

Approach for all Pilots 


Display 

Concept 

Number of 
Data Points 

Subset 

1 

2 

SVS 

19 

0.17 


PBG 

19 

0.19 


SCFD 

18 


0.43 

Significance 


0.655 

1.000 
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Figure 17: VMC-Like Approach, Localizer Error, By Display Concept and Pilot Group 
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4.6.4 ILS approach, Segment 3, no VFR pilots 

Results for localizer error were analyzed without the VFR pilot data for Segment 3 (glideslope and 
localizer tracking), enabling a more direct comparison of the four display concepts within current IMC 
operations. Results from an ANOVA revealed that display concept was highly significant 
(F(3,56)=38.847, p<0.01) and that pilot type (i.e., IFR or Fl-IFR) was not significant for localizer error. 

In addition, there were no significant (p>0.05) interactions between pilot type and display concept for this 
measure. Subsequent post-hoc analysis produced two unique subsets, as shown in Table 47. The 
advanced display concepts (i.e., PBG, SVS, and SCFD) all produced significantly lower localizer error 
than the BRD display concept. This statistically significant result is visually supported by Figure 16, 
which shows the improvement in localizer error for the advanced display concepts. The average localizer 
error for the advanced display concepts was approximately one-fifth the result for the BRD display 
concept. This substantial decrease in localizer error could possibly enable meaningful decreases in the 
obstacle clearance provisions required for ILS approaches due to flight technical error considerations. 

The range of localizer error for the advanced display concept subset was 0.03 dots. Note that the baseline 
minimum required level of localizer error as defined by the IFR and H-IFR pilot data for the ILS BRD 
test herein is 0.39 dots. 

Table 47: Mean Localizer Error (dots) SNK results for the ILS Approach for the IFR and H-IFR 

Pilots 


Display 

Concept 

Number of 
Data Points 

Subset 

1 

2 

PBG 

10 

0.07 


SVS 

15 

0.09 


SCFD 

12 

0.10 


BRD 

19 


0.39* 

Significance 


0.82 

1.00 


*Note: This value is used herein as the baseline performance for an ILS Approach 
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Figure 18: Localizer Error, Segment 3 for ILS and VMC-Like Approaches, All Pilots 


4.6.5 ILS and VMC-Like approach analysis, Segment 3, all pilots 

Figures 16 and 17 show a visual comparison of localizer error results for the ILS and VMC-like 
approaches, respectively. Note that in Figure 17 a large increase in localizer error is shown for the SCFD 
display concept. This large increase in localizer error was due to the turning nature of the VMC-like 
approach and the general lack of turn anticipation of the SCFD display concept (even though desired 
flight path information was provided on the ND). An ANOVA was conducted on the localizer error data 
for all pilots for Segment 3 of the ILS and VMC-like approaches to compare and contrast each advanced 
display concept as well as to assess the capabilities of advanced display concepts to support advanced 
IMC operations. The ANOVA analysis indicated that approach (F(l, 1 1 1)=56.034, p<0.01) and display 
concept (F(2,l 1 1)=15.733, p<0.01) were highly significant and pilot type was not significant for this 
measure. The lack of effect due to pilot type is important and indicates that all types of pilots performed 
the ILS and VMC-like approaches similarly with the advanced display concepts with respect to localizer 
error. This result was not the case for the ILS evaluations of the BRD display concept, as shown in 
Figure 16. There was also a significant interaction between display concept and approach 
(F(2,l 1 1)=12.071, p<0.01) for localizer error. As can be seen in Figures 16, and 17 and Table 48, 
localizer error results for the ILS approach were lower than for the VMC-like approach. 
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Table 48: Mean and Standard Deviation for Localizer Error (dots) for the ILS and VMC-Like 

Approach, All Pilots 


Approach 

Mean 

Standard 

Deviation 

ILS 

0.09 

0.04 

VMC-Like 

0.34 

0.30 


Figures 16 and 17 and Table 49 also show the effect of display concept for the ILS and VMC-like 
approaches. It can be seen that the higher (worst) results were for the SCFD. Results for the SCFD were 
approximately twice as large as those for the PBG and SVS display concepts. This result was most likely 
due to the lack of turn anticipation for the SCFD display concept that was an issue for Segment 3 of the 
VMC-like approach and Segment 2 of the ILS approach. Frequently, EPs would overshoot the base leg 
and/or final approach and in doing so they exceeded the IFR PTS criteria. 

Table 49: Mean Localizer Error (dots) SNK results, ILS and VMC-Like Approach for all Pilots 


Display 

Concept 

Number of 
Data Points 

Subset 

1 

2 

SVS 

40 

0.14 


PBG 

35 

0.16 


SCFD 

36 


0.34 

Significance 


0.623 

1.000 


The interaction between approach and display concept can be seen by comparing Figures 16 and 17 
and Table 50. Mean localizer error increased for all display concepts for the VMC-like approach as 
compared to the ILS approach. However, a much larger increase was experienced for the SCFD display 
concept than for the PBG or SVS display concepts. Localizer error increased by factors of 5.2, 2.9, and 
2.3 for the SCFD, PBG, and SVS display concepts, respectively, for the VMC-like approach as compared 
to the ILS approach. As stated previously, the lack of turn anticipation of the SCFD appears to be the 
reason for the much larger increase in localizer error for this display concept, yielding results that 
exceeded those generated by IFR and H-IFR pilots flying the BRD display concept for the ILS approach 
(i.e., 0.39 dots). 

Increases in localizer error due to approach type differences, as described by the ratio of localizer 
error for the VMC-like to ILS approach, were much larger than lateral flight path error under the same 
conditions. This discrepancy between localizer error and lateral flight path error results was due to the 
fact that there were two turns included in the VMC-like approach occurring at different ranges to 
touchdown. In essence, localizer error can be described as range-weighted lateral flight path error with 
more emphasis placed on errors closer to touchdown. Lateral flight path errors during the turn to final for 
the VMC-Like approach generated larger localizer errors than for the turn to base leg. 

Figures 16 and 17 and Table 49 indicate that, for the VMC-like approach, the increase in localizer 
error for the SVS display concept was smaller than for the PBG display concept. While this was not 
statistically significant, it does suggest that the increase of SA provided by the SVS terrain may actually 
help pilots negotiate the turns of the VMC-like approach. Overall, localizer error results would suggest 
that the VMC-like approach is feasible, but only for the PBG and SVS display concepts. 
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Table 50: Mean and Standard Deviation Results for Localizer Error (dots) for the ILS and VMC- 

Like Approach for all Pilots 


Display 

Concept 

Approach 

Mean 

Standard 

Deviation 

SCFD 

ILS 

0.113 

0.047 


VMC-Like 

0.584 

0.421 

PBG 

ILS 

0.079 

0.026 


VMC-Like 

0.233 

0.085 

SVS 

ILS 

0.089 

0.033 


VMC-Like 

0.202 

0.084 


4. 6. 6 Test condition analysis, Segment 3 

An ANOVA was performed on localizer error for TC with post-hoc results presented in Table 5 1 . 
Test Condition was revealed to be highly significant for localizer error results (F(12,220)=37.76, p<0.01). 
Post-hoc analysis only generated two unique subsets. The highest (worst) subset was TC 1, VFR 
approach with BRD display concept, which had significantly higher localizer error than all other display 
concept and approach combinations evaluated. This result is again not surprising since pilots used only 
the out-the-window view for navigation. There were no significant localizer error differences among the 
other TCs and those mean localizer errors covered a range from 0.584 dots (TC 9) to 0.079 dots (TC 7). 
TC 7 (ILS approach with PBG display concept) localizer error was only 13.5% of the localizer error for 
TC 9 (VMC-like approach with SCFD display concept). Comparing TC 5 (ILS approach with BRD 
display concept) with the other TCs shows that all evaluations by all pilots of the advanced display 
concepts resulted in lower localizer error for the ILS and VMC-like approaches, except for TC 9 (VMC- 
like approach with SCFD display concept). This result, again, is probably due to the lack of turn 
anticipation of the SCFD. Another substantial result was that even though IMC evaluations of the PBG 
and SVS display concept included VFR pilot data (TCs 7, 8, 10, and 1 1), the resulting localizer error was 
less than for the BRD display concept for the ILS with only IFR and H-IFR pilot data. Results for only 
the VFR pilots for the SVS display concept evaluations for the ILS and VMC-like approaches were also 
less than TC 5. The localizer error data suggest that advanced IMC operations, such as the VMC-like 
approach, are feasible with the PBG and SVS advanced display concepts tested, but not for the SCFD 
display concept. 
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Table 51: Mean Localizer Error (dots) SNK results for Test Condition 


Test Condition 

TC 

# 

Number of 
Data Points 

Subset 

1 

2 

ILS Approach + PBG 

7 

16 

0.079 


1LS Approach + SVS 

8 

21 

0.089 


ILS Approach + SVS (VFR pilots) 

12 

6 

0.092 


ILS Approach + SCFD 

6 

18 

0.113 


VMC-Like Approach + SVS 

11 

19 

0.202 


VFR Approach + PBG 

3 

18 

0.226 


VMC-Like Approach + PBG 

10 

19 

0.233 


VMC-Like Approach + SVS (VFR pilots) 

13 

6 

0.239 


VFR Approach + SVS 

4 

18 

0.260 


ILS Approach + BRD (no VFR pilots) 

5 

19 

0.391 


VFR Approach + SCFD 

3 

19 

0.536 


VMC-Like Approach + SCFD 

9 

18 

0.584 


VFR Approach + BRD 

1 

24 


4.036 

Significance 



0.888 

1.000 
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4.7 Glideslope Error 

Comparison of glideslope error results for the various display concepts and approaches is also 
valuable in a similar manner to localizer error (i.e., linear vertical error comparisons favor the advanced 
display concepts since their guidance cues are not affected by range to touchdown). In addition, 
glideslope error analysis and comparisons are valuable due to the linkages with traditional IMC 
operations. Glideslope error results for the four display concepts and two simulated-IMC approaches are 
presented in Figures 19, 20, 21, and 22 and Table 52. Note that the glideslope error results for the VFR 
approach are not plotted in Figure 22 for plotting clarity reasons. 


Table 52: Glideslope Error (dots) Descriptive Statistics for all Pilots 


Display 

Concept 

Approach 

Mean 

Standard 

Deviation 

Number of 
Data Points 

BRD 

VFR 

1.24 

0.88 

24 


1LS 

0.89 

0.66 

30 


Total 

1.05 

0.78 

54 

SCFD 

VFR 

0.32 

0.25 

19 


1LS 

0.16 

0.07 

18 


VMC-Like 

0.30 

0.17 

18 


Total 

0.26 

0.19 

55 

PBG 

VFR 

0.33 

0.14 

18 


1LS 

0.24 

0.11 

16 


VMC-Like 

0.42 

0.23 

19 


Total 

0.34 

0.18 

53 

svs 

VFR 

0.47 

0.38 

18 


1LS 

0.21 

0.09 

21 


VMC-Like 

0.32 

0.11 

19 


Total 

0.33 

0.25 

58 

Total 

VFR 

0.63 

0.66 

79 


1LS 

0.45 

0.52 

85 


VMC-Like 

0.35 

0.18 

56 


Total 

0.49 

0.53 

220 


4. 7.1 VFR approach, all pilots 

As can be seen in Figure 19 and Table 52, the VFR approach evaluation of the BRD display concept 
resulted in larger glideslope error values compared to the advanced display concepts. This result is most 
likely due to the nature of the EPs flight path navigation being driven totally by the out-the-window view 
for the BRD display concept along with altimeter checks at the turn to final from the base leg. Altimeter 
information included in the BRD display concept helped to keep the mean glideslope error to only 1.24 
dots. The apparent increases in glideslope error with decreasing range to touchdown was due to the fact 
that vertical flight path error remained relatively constant throughout the VFR approaches with the 
advanced display concepts as range decreased. An ANOVA conducted on the glideslope error results for 
the main effects of display concept and pilot type revealed highly significant results for display concept 
(F(3,79)=21.5, p<0.01) and non-significant results for pilot type (p>0.05). Subsequent post-hoc analysis 
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resulted in two subsets, as shown in Table 53. As expected, the BRD was the sole member of the worst 
subset, and the advanced display concepts comprised the best subset. Note that glideslope error for the 
SVS display concept was 0. 14 dots worse than the PBG. This may be attributed to the evaluation pilots 
frequently comparing the outside visual scene to that provided by the SVS terrain and paying slightly less 
attention to the guidance cues. 

Table 53: Mean Glideslope Error (dots) SNK results for Display Concept for the VFR Approach 

for all Pilots 


Display 

Concept 

Number of 
Data Points 

Subset 

1 

2 

SCFD 

19 

0.32 


PBG 

18 

0.33 


SVS 

18 

0.47 


BRD 

24 


1.235 

Significance 


0.55 

1.000 
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Figure 19: VFR Approach, Glideslope Error, By Display Concept and Pilot Group 
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4. 7.2 ILS approach, all pilots 

For the ILS approach for all pilots, the glideslope error decreased for all display concepts compared 
to the VFR approach, as can be seen by comparing Table 54 with Table 53 and by comparing Figures 19 
and 20. Figure 20 shows glideslope error for the four display concepts for all pilots for the ILS approach. 
Note that glideslope error results for the BRD display concept occasionally exceeded the IFR PTS 
criteria. While most of the evaluations that exceeded the IFR PTS criteria were for VFR pilots flying 
with the BRD display concept, one was for an H-IFR EP. Figure 20 also shows that results for the 
advanced display concepts were all well within the IFR PTS criteria. A slight decrease in glideslope error 
is also visible for the SVS compared to the PBG display concept, potentially indicating the effect of SVS 
terrain to improve glideslope error. An ANOVA conducted on the glideslope error results for the main 
effects of display concept and pilot type revealed highly significant results for display concept 
(F(3,85)=20.8, p<0.01) and non-significant results for pilot type (p>0.05). There were no significant 
interactions. The lowest glideslope error was for the SCFD display concept (0.16 dots) and the highest 
was for the BRD display concept (0.89 dots). The advanced display concepts had a range of glideslope 
error results of approximately 0. 16 to 0.24 dots and were included in the lowest (best) subset, while the 
BRD was the only member of the higher (worst) subset. This result was expected due to the integrated 
nature of the advanced display concepts along with their guidance cues. The inclusion of the VFR pilot 
data also degrades the results for the BRD display concept. 

Table 54: Mean Glideslope Error (dots) SNK results for Display Concept, ILS Approach for all 

Pilots 


Display 

Concept 

Number of 
Data Points 

Subset 

1 

2 

SCFD 

18 

0.16 


SVS 

21 

0.21 


PBG 

16 

0.24 


BRD 

30 


0.89 

Significance 


.779 

1.000 


While results were not statistically significant for pilot type, post-hoc analysis did produce two 
subsets as shown below in Table 55. There was a 0.27 dot difference in glideslope error for the VFR 
pilots compared to the mean for the H-IFR and IFR pilots for all displays tested. Figure 20 shows that 
much of the difference in glideslope error due to pilot type was likely due to the BRD display concept. 

Table 55: Mean Glideslope Error (dots) SNK results for Display Concept, VFR Approach for all 

Pilots 


Pilot Type 

Number of 
Data Points 

Subset 

1 

2 

H-IFR 

27 

0.32 


IFR 

29 

0.39 


VFR 

29 


0.62 

Significance 


0.475 

1.000 
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Figure 20: ILS Approach, Glideslope Error, By Display Concept and Pilot Group 
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4. 7.3 VMC-Like approach 

Unlike localizer error results, Figure 21 shows that glideslope error was fairly similar for all three 
advanced display concepts for the VMC-like approach. Flowever, the worst results were for the PBG 
display concept with several excursions above the PTS criteria evident in Figure 21 during the final 
approach leg (approximately 1.8 nm to the 0.66 nm). An ANOVA conducted on the glideslope error data 
for the VMC-like approach for the main effects of display concept and pilot type revealed significant 
results for display concept (F(2,56)=3.52, p=0.04) and non-significant results for pilot type (p>0.05). 
There was no significant interaction between display concept and pilot type. Subsequent post-hoc 
analysis resulted in two subsets, as shown in Table 56. Flowever, in this case the highest (worst) subset- 
group was comprised of only the PBG display concept, with the SVS display concept being grouped with 
the SCFD in the lowest (best) subset. Part of this result is due to the improved vertical flight path error 
control of the single-cue flight director compared to the PUTS guidance discussed previously. Flowever, 
for glideslope results, the inclusion of the SVS terrain with the PUTS guidance significantly reduced 
glideslope error by 0. 10 dots compared to the PBG display concept. This reduction of the glideslope error 
resulted in the SVS display concept grouping with the SCFD. The fact that this is significant here as 
opposed to the vertical flight path error is likely due to the range-weighted nature of glideslope error 
combined with the increased presence of the SVS terrain with runway models on final approach. 

Vertical flight path errors generate larger glideslope errors as range to the runway touchdown zone 
decreases. As a result glideslope error is more sensitive to vertical error during the last several miles 
during final approach than at the beginning of the approach. If results for vertical flight path error varied 
similarly during the approach for all display concepts, then statistical analysis results for vertical flight 
path and glideslope errors would produce similar SNK groupings. However, as shown in Tables 84 
(vertical flight path error) and 56 (glideslope error) the SVS display grouped in the worst group for 
vertical flight path error and in the best group for glideslope error. In order for this to occur, vertical 
flight path error results had to decrease during the approach for the SVS display concept more than the 
SCFD and PBG display concepts. The distribution of vertical flight path and glideslope errors can be 
seen in Figures 33 and 21. From Figure 33, it can be observed that vertical flight path error for the SVS 
display concept appears to become similar to the SCFD as range to touchdown decreases. Results for the 
PBG display concept appear to remain constant and even increase during the last 1 .5 nm of the approach. 
From Figure 21, it can be seen that the glideslope error results for the SVS display closely resemble those 
for the SCFD and remain within IFR PTS criteria throughout the approach. Figure 2 1 also reveals that 
results for the PBG display concept not only increased but exceeded IFR PTS criteria during the last 1.5 
nm of the approach. It is considered that the ability to integrate information is different for the VFR, IFR, 
and H-IFR pilots groups. Pilot training associated with achieving an IFR pilot rating combined with 
thousands of hours of actual IMC flight time develops the ability to integrate information provided by 
separate displays. While PFDs provide information in a more integrated manner than the BRD, the 
background SVS terrain further facilitates the pilots’ ability to mentally process the data. Results for pilot 
type were not significant for this analysis, however, from Figure 21 it can be seen that the VFR pilots and 
one of the IFR pilots were having difficulty remaining within the PTS criteria with the PBG display 
concept. Glideslope error results for the SVS display concept reveal similar performance for all pilot 
groups. Again, it is believed that the SVS terrain serves as an integrator of other information allowing all 
pilot groups to achieve not only superior situation awareness but also improved performance. 
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Table 56: Mean Glideslope Error (dots) SNK results for Display Concept, VMC-Like Approach 

for all Pilots 


Display 

Concept 

Number of 
Data Points 

Subset 

1 

2 

SCFD 

18 

0.30 


svs 

19 

0.32 


PBG 

19 


0.42 

Significance 


0.675 

1.000 
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Figure 21: VMC-Like Approach, Glideslope Error, By Display Concept and Pilot Group 
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4. 7.4 ILS approach, Segment 3, no VFR pilots 

Results for glideslope error were analyzed without the VFR pilot data, enabling a more direct 
comparison of the four display concepts within current IMC operations. Results from an ANOVA 
revealed that display concept was highly significant (F(3,56)=16.68, p<0.01) and that pilot type (i.e., IFR 
or Fl-IFR) was not significant (p>0.05). In addition, there were no significant interactions between pilot 
type and display concept. Subsequent post-hoc analysis produced two unique subsets, as shown in Table 
57. With a mean of 0.76 dots error, the BRD display concept had significantly higher glideslope error 
than the advanced display concepts. There were no significant glideslope error differences among the 
SCFD, SVS, and PBG concepts. The average mean glideslope error for the advanced display concepts 
was approximately one quarter of glideslope error for the BRD display concept. These results are 
visually supported by Figure 20. Again, as with localizer error, the reduction in glideslope error for the 
advanced display concepts may support reductions in the amount of airspace required for ILS approaches 
with the advanced display concepts. As discussed previously for vertical flight path error results, while 
other limitations may apply, obstacle clearance margins could be reduced by a factor of four. Note that 
the baseline minimum required level of glideslope error defined by the IFR and H-IFR BRD pilot data for 
the test herein is 0.76 dots. 

Table 57: Mean Glideslope Error (dots) SNK results for the ILS Approach, IFR and H-IFR Pilots 


Display 

Concept 

Number of 
Data Points 

Subset 

1 

2 

SCFD 

12 

0.14 


SVS 

15 

0.20 


PBG 

10 

0.21 


BRD 

19 


0.76* 

Significance 


0.759 

1.000 


*Note: This value is used herein as the baseline performance for an ILS Approach 
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Figure 22: Glideslope Error For Segment 3, for Approaches 2 and 3, All Pilots 


4. 7.5 ILS and VMC-Like approach analysis, Segment 3, all pilots 

Comparison of Figures 20 and 21 provides a visual indication of the effect of approach and display 
concept for the advanced display concepts and the ILS and VMC-like approaches. From these figures, it 
can be seen that glideslope error increased for all the advanced display concepts, with the largest increase 
for the PBG display concept. While the FIITS guidance concept provided a good blending of lateral and 
vertical flight path error performance overall, the VMC-like evaluation of the PBG display concept 
resulted in higher levels of glideslope error with several evaluations exceeding the IFR PTS criteria. 
Figure 21 appears to indicate that the presence of the SVS background limited the increase in glideslope 
error for the SVS display concept while flying the VMC-like approach from 2.0 nm to the end of the 
approach. 

An ANOVA was performed on the glideslope error data for all pilots for segment 3, for the ILS and 
VMC-like approaches for the main effects of pilot type, display concept, and approach to assess the 
capabilities of advanced display concepts to support advanced IMC operations. The ANOVA analysis 
revealed that approach (F (1,11 1)=60.084, p<0.01) was highly significant and that display concept 
(F(2,l 1 1)=4.729, p=0.01 1) and pilot type (F(2,l 1 1)=3.220, p=0.045) were also significant but at lower 
levels. Unlike most of the previous analysis for the ILS and VMC-like approaches, pilot type was 
significant for this analysis. There were no significant interactions. Subsequent post-hoc analysis 
generated two unique subsets for display concept, as shown in Table 58, but no subsets for pilot type. 

The PBG display concept had significantly higher glideslope error compared to the SVS and SCFD 
display concepts. There were no significant glideslope error differences between the SVS and SCFD 
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concepts. This result reflects the inclusion of SVS terrain information in the SVS display concept, which 
appears to have significantly reduced glideslope error compared to the PBG display concept. Much of 
this result was due to the superior performance, or lack of substantial degradation of performance, for the 
VFR pilots for the VMC-like approach with the SVS display concept. The integration of SVS terrain 
with HITS guidance symbology most likely produced an environment where the VFR evaluation pilots 
were able to perform the VMC-like approach as well as the other pilot groups. 

During initial system checkout, several pilots evaluated the research displays during the checkout 
flights to determine if the flight test implementation was similar to what was implemented in the 
simulation. Pilots were asked to provide qualitative inputs regarding flight path tracking error 
performance and workload. Flight path and angular errors gains of the SCFD were adjusted to achieve 
representative performance. The gains used for the research flights are listed in Appendix A. 

As implemented for this flight test, the SCFD provided significantly improved glideslope error 
results compared to the PBG display concept. This was due to the single-cue flight director drive 
equations combined with the prominent display of pitch attitude commands on the SCFD display that 
provided a large emphasis on vertical error control. Results for glideslope error for the PBG and SVS 
display concepts could potentially be improved through adjustment of the tadpole guidance cue (see 
Appendix B for description of guidance cue). For this study, the tadpole guidance cue was located along 
the desired flight path 15 seconds in front of ownship. This timing parameter was established from work 
with large transport-type aircraft (Reference 8) and could potentially be reduced for GA applications. 
Having the guidance cue located closer to ownship would help to reduce flight path error. Overall, the 
guidance cue of the HITS symbology was designed to provide a good blending of lateral and vertical 
error control, which it accomplished. It provided much better results than the SCFD, when considering 
combined lateral and vertical errors, SA, and TLX, and DRR results. 

Table 58: Mean Glideslope Error (dots) SNK results for the ILS and VMC-Like Approaches for all 

Pilots 


Display 

Concept 

Number of 
Data Points 

Subset 

1 

2 

SCFD 

36 

0.28 


SVS 

40 

0.31 


PBG 

35 


0.41 

Significance 


0.334 

1.000 


The interactions between display concept and pilot type are illustrated in Figures 20 and 21 and 
Table 59. It can be seen that the VFR pilots performed worse in terms of glideslope error than the IFR 
and H-IFR pilots for all display concepts with VFR pilot data being worse than the data from the other 
two pilot groups. However, the level of increase of glideslope error for the VFR pilots compared to the 
average of the means of the IFR and H-IFR pilots data was not the same for all display concepts, with 
increases of 0. 12, 0.09, and 0.04 dots for the SCFD, PBG, and SVS display concepts, respectively. The 
increase in glideslope error for the VFR pilots for the SVS display concept was only approximately one 
third to one half of the increase for the SCFD and PBG display concepts. This lower level of increase for 
the SVS display concept is likely due to the presence of the SVS terrain providing the VFR pilots with SA 
unavailable for the other advanced display concepts due to the simulated-IMC test conditions. 
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Table 59: Statistical Interaction Results for Glideslope Error (dots) for Display Concept and Pilot 

Type for the ILS and VMC-Like approaches 


Display 

Concept 

Pilot Type 

Mean 

Standard 

Deviation 

Number of 
Data Points 

SCFD 

VFR 

0.36 

0.21 

12 


IFR 

0.25 

0.22 

12 


H-IFR 

0.23 

0.17 

12 


Total 

0.28 

0.20 

36 

PBG 

VFR 

0.47 

0.39 

12 


IFR 

0.41 

0.25 

10 


H-IFR 

0.36 

0.20 

13 


Total 

0.41 

0.29 

35 

SVS 

VFR 

0.34 

0.17 

12 


IFR 

0.28 

0.16 

14 


H-IFR 

0.32 

0.14 

14 


Total 

0.31 

0.15 

40 

Total 

VFR 

0.39 

0.27 

36 


IFR 

0.31 

0.21 

36 


H-IFR 

0.30 

0.18 

39 


Total 

0.33 

0.23 

111 


Table 60 shows the glideslope error results for the advanced display concepts for each IMC 
approach. As stated previously, the largest increase in glideslope error was for the PBG display concept, 
which experienced a 0.29 dot increase for the VMC-like approach vs. the ILS approach. The smallest 
increase in glideslope error was for the SVS display concept with a 0. 19 dot increase for the VMC-like 
approach. This positive result for the SVS display concept glideslope error was primarily because the 
VFR pilots performed much better using the SVS display concept for the VMC-like approach than they 
did using other display concepts. Overall, glideslope errors for all display concepts for the VMC-like 
approach were less than the ILS evaluation of the BRD display concept for the IFR and H-IFR pilots ( i.e., 
0.76 dots), indicating that the VMC-like approach may be feasible for these display concepts. 


88 




Table 60: Statistical Interaction Glideslope Error (dots) Data for Display Concept and Approach 

for the ILS and VMC-Like Approaches 


Display 

Concept 

Approach 

Mean 

Standard 

Deviation 

Number of 
Data Points 

SCFD 

ILS 

0.16 

0.08 

18 


VMC-Like 

0.39 

0.23 

18 


Total 

0.28 

0.20 

36 

PBG 

ILS 

0.25 

0.11 

16 


VMC-Like 

0.55 

0.33 

19 


Total 

0.41 

0.29 

35 

SVS 

ILS 

0.22 

0.09 

21 


VMC-Like 

0.41 

0.14 

19 


Total 

0.31 

0.15 

40 

Total 

ILS 

0.21 

0.10 

55 


VMC-Like 

0.45 

0.25 

56 


Total 

0.33 

0.23 

ill 


4. 7.6 Test condition analysis, Segment 3 

An ANOVA was performed on glideslope error for TC with post-hoc results presented in Table 61. 
TC was highly significant for glideslope error results (F(12,220)=14.266, p<0.01). Post-hoc analysis of 
the glideslope error generated two unique subsets. The VFR approach with the BRD (TC 1) had 
significantly higher glideslope error than all other approach and display concept combinations tested. 

Most likely, this result is because the pilots used the out-the-window method of navigation. No 
significant differences were seen in this test. All of the other TCs were grouped into one subset, and they 
had a range of mean RMS glideslope errors of approximately 0.6 dots (from 0.163 dots to 0.760 dots). 

TC 6 (ILS approach with SCFD display concept) yielded the lowest glideslope error of 0.16 dots. Test 
Condition 5 (ILS approach with the BRD display concept, no VFR pilots) had the highest glideslope error 
of 0.76 dots for this subset, which was 4.7 times larger than the result for TC 6. Overall, all IMC 
evaluations of the advanced display concepts (TC 6, 7, 8, 9, 10, and 1 1) yielded lower glideslope errors 
than TC 5, including the IMC VFR-only pilot evaluations of the SVS display concept (TC 12 and 13). 
This result indicates that the advanced display concepts provided all pilot types the ability to control 
glideslope error as well or better than the current baseline standard (BRD display concept for the ILS 
approach for IFR and H-IFR pilots) employed in this test. The VFR approach results for the advanced 
display concepts (i.e., TC 2, 3, and 4) were also as good or better than TC 5, even though EPs were 
performing out-the-window navigation combined with cross checking from their displays. 
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Table 61: Mean Glideslope Error (dots) SNK results for Test Condition 


Test Condition 

TC# 

Number of 
Data Points 

Subset 

1 

2 

1LS Approach + SCFD 

6 

18 

0.163 


1LS Approach + SVS 

8 

21 

0.220 


ILS Approach + PBG 

7 

16 

0.254 


1LS Approach + SVS (VFR pilots) 

12 

6 

0.272 


VMC-Like Approach + SCFD 

9 

18 

0.388 


VMC-Like Approach + SVS (VFR pilots) 

13 

6 

0.406 


VMC-Like Approach + SVS 

11 

19 

0.414 


VFR Approach + SCFD 

2 

19 

0.416 


VFR Approach + PBG 

3 

18 

0.426 


VMC-Like Approach + PBG 

10 

19 

0.546 


VFR Approach + SVS 

4 

18 

0.629 


ILS Approach + BRD (no VFR pilots) 

5 

19 

0.760 


VFR Approach + BRD 

1 

24 


1.840 

Significance 



0.177 

1.000 
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4.8 Indicated Airspeed Error 

Airspeed error is valuable to describe the differences between the display concepts, approaches, and 
pilot types. It can be viewed as a consistent measure of pilot awareness, since the task of controlling 
airspeed is similar from one approach to the next. Airspeed results are presented in Figures 23, 24, and 25 
for the VFR, ILS, and VMC-like approaches, respectively. Figure 26 along with Tables 62 and 63 
present mean RMS airspeed error results. Recall that for Segment 2, pilots were asked to maintain 100 
kts and then decelerate and change speed to 90 kts for Segment 3. Due to limitations in data processing, 
airspeed errors are only presented separately for Segment 2 and Segment 3 of the approaches evaluated. 


Table 62: IAS Error (kts) Descriptive Statistics for Segment 2 for all Pilots 


Display 

Concept 

Approach 

Mean 

Standard 

Deviation 

Number of 
Data Points 

BRD 

VFR 

4.6 

1.5 

24 


ILS 

5.0 

2.3 

30 


Total 

4.9 

2.0 

54 

SCFD 

VFR 

4.7 

1.9 

19 


ILS 

5.8 

2.9 

18 


VMC-Like 

4.6 

2.4 

18 


Total 

5.0 

2.4 

55 

PBG 

VFR 

3.9 

2.3 

18 


ILS 

4.4 

3.2 

16 


VMC-Like 

4.5 

2.1 

19 


Total 

4.3 

2.5 

53 

svs 

VFR 

3.6 

1.9 

18 


ILS 

4.5 

1.6 

21 


VMC-Like 

4.6 

2.6 

19 


Total 

4.3 

2.1 

58 

Total 

VFR 

4.3 

1.9 

79 


ILS 

5.0 

2.5 

85 


VMC-Like 

4.6 

2.3 

56 


Total 

4.6 

2.4 

220 
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Table 63: IAS Error (kts) Descriptive Statistics for Segment 3 for all Pilots 


Display 

Concept 

Approach 

Mean 

Standard 

Deviation 

Number of 
Data Points 

BRD 

VFR 

3.6 

1.4 

24 


ILS 

5.9 

2.5 

30 


Total 

4.9 

2.3 

54 

SCFD 

VFR 

5.6 

2.2 

19 


ILS 

6.3 

2.9 

18 


VMC-Like 

6.1 

2.2 

18 


Total 

6.0 

2.4 

55 

PBG 

VFR 

5.0 

1.6 

18 


ILS 

5.6 

2.1 

16 


VMC-Like 

6.1 

2.9 

19 


Total 

5.6 

2.3 

53 

SVS 

VFR 

5.7 

2.1 

18 


ILS 

5.6 

2.1 

21 


VMC-Like 

5.4 

1.6 

19 


Total 

5.6 

1.9 

58 

Total 

VFR 

4.9 

2.0 

79 


ILS 

5.8 

2.4 

85 


VMC-Like 

5.9 

2.3 

56 


Total 

5.5 

2.3 

220 


4.8.1 VFR approach all pilots, S2 and 3 

As can be seen from Figure 23 and Tables 62 and 63, the difference in airspeed error for the BRD 
and the advanced display concepts was much less pronounced than for other data measures discussed 
previously. The BRD display concept produced less error than the other display concepts as shown in 
Figure 23 where the pilots were able to maintain airspeed well within the +/-10 kts PTS criterion most of 
the time. The difference in IAS error between the BRD and advanced display concepts was likely related 
to the manner in which airspeed was presented to the pilot and also the content and use of that 
information for the advanced display concepts. For the BRD display concept, airspeed was a simple 
gauge that pilots looked at for current airspeed (position) and trend information (rate of movement). For 
the advanced display concepts, current airspeed was presented digitally on the airspeed tape, which also 
provided airspeed rate information (tape movements, acceleration trend line). The acceleration arrow on 
the velocity vector also provided airspeed rate information for the PBG and SVS display concepts (see 
Figure 6). ANOVAs conducted on Segment 2 and 3 IAS errors for the main effects of display concept 
and pilot type resulted in non-significant results for Segment 2, but highly significant results for display 
concept (F(3,79)=6.65, p<0.01) for Segment 3. Subsequent post-hoc results are shown in Table 64, where 
it can be seen that two unique subsets were formed. The lowest (best) subset was comprised only of the 
BRD display concept with the advanced display concepts being included within the highest (worst) 
subset. IAS error was 1.4 kts less for the BRD display concept compared to the PBG, and 2.2 kts less 
compared to the SVS display concept. These results show that the round-dial method of airspeed 
presentation may be superior to the integrated digital readouts and tapes of the advanced display concepts 
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for the VFR approach where a relatively low number of disparate pieces of information need to be 
integrated (i.e., out-the-window-view, airspeed, and altitude). 


Table 64: Mean IAS Error (kts) SNK results for the VFR Approach, Segment 3 for all Pilots 


Display 

Concept 

Number of 
Data Points 

Subset 

1 

2 

[3RD 

24 

3.6 


PBG 

18 


5.0 

SCFD 

19 


5.6 

svs 

18 


5.7 

Significance 


1.000 

0.37 
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Figure 23: VFR Approach, IAS Performance for All Pilots and Display Concepts 
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4.8.2 ILS approach, all pilots, Segments 2 and 3 

Figures 23 and 24 and Tables 62 and 63 provide a visual comparison of airspeed control 
performance for the VFR and ILS approaches, respectively. For the ILS approach for all pilots, IAS error 
increased for all display concepts except for the SVS display concept compared to the VFR approach. 

The largest increase was for the BRD display concept with an increase of 2.3 kts (Segment 3). The 
advanced display concepts led to much lower increases of IAS error for the ILS approach compared to the 
VFR approach. IAS error increased 0.7 kts and 0.4 kts for the SCFD and PBG display concepts, 
respectively. IAS error actually decreased for the SVS display concept for the ILS approach, but only 
slightly, 0. 1 kts. The fact that IAS error increased much more for the BRD display concept than for the 
advanced display concepts indicates that while the IAS gauge potentially provided improved airspeed 
awareness compared to the integrated airspeed tapes of the advanced display concepts, it was integrated 
within the BRD display concept, resulting in increased overall workload. This resulted in a larger effect 
of approach in IAS error for this display concept. An ANOVA conducted on the IAS error for Segments 
2 and 3 revealed that pilot type was significant for Segment 2 (F(2,85)=4.76, p=0.012) and Segment 3 
(F(2,85)=4.45, p=0.016) and non-significant for display concept. The interaction between display 
concept and pilot type was not statistically significant. Tables 65 and 66 show the results from the SNK 
post-hoc analysis and that the VFR pilots had significantly worse IAS error than the IFR or H-IFR pilots. 
While the results presented in Tables 65 and 66 indicate a significant difference between the VFR and 
IFR/H-IFR pilots, the increased IAS error for the VFR pilots was not the same for all display concepts. 
Table 67 provides the IAS error increase for the VFR pilots compared to the average of the IFR and H- 
IFR pilots for all of the display concepts. From Table 67, it can be seen that the increase of IAS error for 
the VFR pilots was lower for the SVS display concept compared to all the others, which could reflect the 
lower workload and increased situation awareness of the SVS display concept. While the statistical 
analysis revealed no significant interactions between display concept and pilot type, results provided in 
Table 67 do show a trend in the data that indicates that the degraded VFR-pilot IAS error control was 
mitigated for the SVS display concept. 

Table 65: Mean IAS Error (kts) SNK results for the ILS Approach, Segment 2 for all Pilots 


Pilot Type 

Number of 
Data Points 

Subset 

1 

2 

H-IFR 

27 

3.9 


IFR 

29 

4.9 


VFR 

29 


6.2 

Significance 


0.115 

1.000 


Table 66: Mean IAS Error (kts) SNK results for the ILS Approach, Segment 3 for all Pilots 


Pilot Type 

Number of 
Data Points 

Subset 

1 

2 

IFR 

29 

5.1 


H-IFR 

27 

5.7 


VFR 

29 


6.8 

Significance 


0.264 

1.000 
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Figure 24: ILS Approach, IAS Performance for All Pilots and Display Concepts 
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Table 67: Increase in mean IAS Error (kts) for the VFR Pilots Compared to the Average of the 
IFR and H-IFR Pilots for the IFS Approach, Segments 2 and 3 


Display 

Concept 

Segment 

2 

3 

BRD 

1.9 

1.9 

SCFD 

2.4 

1.0 

PBG 

3.5 

1.6 

SVS 

0.2 

0.9 


4.8.3 ILS approach, Segment 3, no VFR pilots 

Results for IAS error were analyzed without the VFR pilot data, enabling a more direct comparison 
of the four display concepts within current IMC operations. Results from an ANOVA with display 
concept and pilot type as main effects revealed no significant (p>0.05) differences for IAS error for either 
the main effects or their interactions for segment 2 or segment 3. Although not statistically significant, H- 
IFR pilots controlled IAS approximately 1.0 kts better than the IFR pilots for Segments 2 and 3. Table 68 
shows the mean IAS error for the ILS approach without the VFR pilot data. The lowest IAS error was for 
the PBG display concept for both segments and the worst was for the SCFD display concept. Note that 
IAS error for Segment 3 for the BRD display concept was 5.2 kts, which establishes the baseline 
minimum required level for ILS approaches for the data herein. Unlike previous measures, the BRD 
display concept did not produce the worst performance. While the PBG and SVS display concepts were 
within +1-4% of the BRD display concept result, the IAS error for the SCFD was 15% higher than the 
BRD display concept. 

Table 68: Mean IAS Error (kts) for Segment 2 and Segment 3 of the ILS Approach, IFR and H- 

IFR Pilots 


Display 

Concept 

Number of 
Data Points 

Mean 
Segment 2 

Mean 
Segment 3 

PBG 

10 

3.1 

5.0 

BRD 

19 

4.5 

5.2* 

SVS 

15 

4.7 

5.3 

SCFD 

12 

5.0 

6.0 


*Note: This value is used herein as the baseline performance for an ILS Approach 


4.8.4 VMC-Like approach, all pilots, Segments 2 and 3 

An ANOVA conducted on the IAS error results for the main effects of display concept and pilot type 
revealed no statistically significant results for either Segment 2 or Segment 3. Table 69 provides IAS 
error results for the three advanced display concepts for the VMC-like approach. 
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Table 69: IAS Error (kts) for Segment 2 and Segment 3 of the VMC-Like Approach for All Pilots 


Display 

Concept 

Number of 
Data Points 

Mean 
Segment 2 

Mean 
Segment 3 

PBG 

19 

WKBM 

6.1 

SVS 

19 

’dmm 

5.4 

SCFD 

18 

— 

6.1 


4.8.5 ILS and VMC-Like approach analysis, Segment 3, all pilots 

Figure 25 provides airspeed results for the VMC-like approach for the advanced display concepts. 
Figures 24 and 25 provide a visual comparison of the pilots’ ability to control airspeed for the ILS and 
VMC-like approaches, respectively. An ANOVA was conducted on the IAS error data for all pilots for 
Segment 3 of the ILS and VMC-like approaches for the main effects of pilot type, display concept, and 
approach to assess the capabilities of advanced display concepts to support advanced IMC operations. 
There were no significant (p>0.05) main effects or interaction effects for IAS error. However, the highest 
(worst) IAS error for these two approaches was for the VFR pilots (6.3 kts), with the IFR pilots and H- 
IFR pilots generating the same IAS error (5.7 kts). While this result was not statistically or operationally 
significant, it appears to be a trend in the data and reflects pilot training and hours of experience. Overall, 
the VFR pilots’ IAS error was 10% worse than the other two pilot groups. With regard to display 
concept, the worst (largest) IAS error was for the SCFD (6.3 kts) while the best (smallest) was for the 
SVS (5.5 kts). This result is supported by Figure 25, which shows that the airspeed control for the SCFD 
display concept would exceed the PTS criteria more often than for the PBG or SVS display concepts. 

The IAS error for the PBG display concept (5.8 kts) was very close to the SVS display concept. This 
result implies that the SA provided by the SVS terrain, which yields no information regarding airspeed, 
did not affect the pilots’ ability to control airspeed. The SVS display concept decreased IAS error by 
13% compared to the SCFD for the ILS and VMC-like approaches. For all advanced display concepts, 
the average IAS error for each approach (i.e., ILS or VMC-Like) was virtually identical at 5.9 kts. This 
implies that any increase in workload due to the VMC-like approach was not reflected at all in the 
airspeed error data. 

Table 70 provides a description of the IAS errors for the advanced display concepts with the VFR 
pilot data included compared to the ILS BRD display concept baseline (i.e., BRD display concept for the 
ILS approach for the IFR and H-IFR pilots). This comparison is presented in order to discuss the 
capability of the advanced display concepts to accommodate low-time VFR pilots and enable advanced 
IMC approaches. As can be seen in Table 70, the largest increase in the IAS error was for SCFD display 
concept, which was 22.9% and 18.2% higher than the baseline for the ILS and VMC-like approaches, 
respectively. The smallest increase in IAS error was for the SVS display concept, which was only 5.6% 
and 5.4% higher than the baseline error for the ILS and VMC-like approaches, respectively. The fact that 
the SVS display concept was the only one to limit increases in IAS error for VMC-like approach suggests 
that this display concept was the only one of those tested that could potentially enable advanced IMC 
operations for virtually all currently certificated pilots. While none of these results were statistically 
significant, they do show trends in the data that support workload and situation awareness results 
discussed previously. 
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Table 70: Percent IAS Error Increase Compared to ILS Evaluation of BRD Display Concept (IFR 
and H-IFR Pilots) for Segment 3 for ILS and VMC-Like Approaches (all Pilots) 


Display 

Concept 

Approach 

ILS 

VMC-Like 

SCFD 

22.9% 

18.2% 

PBG 

8.3% 

18.2% 

SYS 

5.6% 

5.4% 
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Figure 25: VMC-Like Approach, IAS Performance for all Pilots and Display Concepts 
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Approach 


Figure 26: IAS Error for Segment 3, All Pilots, All Approaches 


4. 8. 6 Test condition analysis, Segment 3 

A subsequent analysis was performed on IAS error based on TC, with post-hoc results presented in 
Table 71. As previously performed for other analyses, VFR pilot data was not included for TC 5 (BRD 
display concept for the ILS approach) in order to provide a more comprehensive measure with strong ties 
to current operations. Test Condition was highly significant for IAS error results (F(12,220)=2.73, 
p<0.01). As can be seen in Table 71, two overlapping subsets were formed from SNK post-hoc analysis. 
The subset with the lowest IAS error was comprised of TC 1 (VFR approach with BRD display concept), 
TC 3 (VFR approach with PBG display concept), and TC 5 (ILS approach with BRD display concept, 

IFR and H-IFR pilots only). The result is not surprising and reflects the low-workload environment of the 
VFR approach and superior presentation of airspeed information to the pilot afforded by the BRD display 
concept. Results for the VFR approach for the PBG display concept being included in the low- and high- 
IAS error groups is a little surprising and is potentially due to the lack of SVS terrain background for this 
display concept. During the research evaluations, pilots would periodically check and verify that the SVS 
terrain inside matched the terrain outside and may have become slightly distracted. This distraction was 
not there for the PBG display concept, since there was no terrain presented on the PFD. TC 5 was also 
included in the low- and high-IAS error groups even though there was a 1.6 kts increase compared to TC 
1. Note that the increase in airspeed error due to outside visibility was smaller for the advanced display 
concepts than for the BRD. The highest IAS error subset was comprised of all the other TCs (i.e., 2 
through 13). As discussed previously, the ILS BRD baseline condition (i.e., TC 5) provided lower 
airspeed error than the other TCs in simulated IMC, but these differences were not significant. The 
difference between TC 5 and SVS IMC evaluations (i.e., TC 8 and 1 1) is smaller compared to differences 
from the other advanced display concepts. It also needs to be recognized that the advanced display 
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concepts included the VFR pilot data, whereas TC 5 did not. Results for the VFR pilot evaluations of the 
SVS display concept for the 1LS and VMC-like approaches were higher than TC 5, but not significantly. 
This implies that advanced IMC operations may be possible for the advanced display concepts. 


Table 71: Mean IAS Error (kts) SNK results for TC 


Test Condition 

TC# 

Number of 
Data Points 

Subset 

1 

2 

VFR Approach + BRD 

1 

24 

3.6 


VFR Approach + PBG 

3 

18 

5.0 

5.0 

ILS Approach + BRD (no VFR pilots) 

5 

19 

5.2 

5.2 

VMC-Like Approach + SVS 

11 

19 


5.4 

ILS Approach + SVS 

8 

21 


5.6 

ILS Approach + PBG 

7 

16 


5.6 

VFR Approach + SCFD 

2 

19 


5.6 

VFR Approach + SVS 

4 

18 


5.7 

VMC-Like Approach + SVS (VFR pilots) 

11 

6 


5.7 

VMC-Like Approach + SCFD 

9 

18 


6.1 

VMC-Like Approach + PBG 

10 

19 


6.1 

ILS Approach + SVS (VFR pilots) 

12 

6 


6.2 

ILS Approach + SCFD 

6 

18 


6.3 

Significance 



0.054 

0.686 


4.9 Lateral Flight-Path Error 

Lateral flight path error results for the four display concepts and three approaches are presented in 
Table 72. Figures 27, 28, and 29 show real-time lateral flight path error for the VFR, 1LS, and VMC-like 
approaches, respectively. Figure 30 illustrates RMS lateral flight path error results for the 1LS and VMC- 
like approaches. Error results for the VFR approach evaluation of the BRD were too large to plot 
conveniently as they were approximately five times greater than other results. 
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Table 72: Lateral Flight-Path Error (ft) Descriptive Statistics for all Pilots 


Display 

Concept 

Approach 

Mean 

Standard 

Deviation 

Number of 
Data Points 

BRD 

VFR 

1096.4 

501.4 

24 


1FS 

624.1 

428.2 

30 


Total 

834.0 

515.4 

54 

SCFD 

VFR 

141.2 

117.1 

19 


IFS 

140.1 

95.5 

18 


VMC-Fike 

138.6 

84.2 

18 


Total 

140.0 

98.4 

55 

PBG 

VFR 

70.2 

36.5 

18 


IFS 

73.8 

35.0 

16 


VMC-Fike 

80.2 

38.2 

19 


Total 

74.9 

36.2 

53 

svs 

VFR 

79.7 

54.2 

18 


IFS 

72.9 

30.7 

21 


VMC-Fike 

70.9 

36.8 

19 


Total 

74.4 

40.6 

58 

Total 

VFR 

401.2 

540.9 

79 | 


IFS 

281.9 

361.8 

85 


VMC-Fike 

95.8 

63.3 

56 ; 


Total 

277.4 

411.6 

220 ! 


4.9.1 VFR approach 

Evaluation pilots were instructed to fly the VFR approach and cross check their out-the-window 
navigation with the guidance and navigation information provided from the ND and PFD. As can be seen 
from Figures 27 and Table 72, lateral flight path error for the VFR approach for the BRD display concept 
was approximately an order of magnitude larger than for the advanced display concepts, which is not an 
unexpected result. An ANOVA performed on the lateral flight path error for the main effects of display 
concept and pilot type resulted in highly significant results for display concept (F(3, 79)=72.0, p<0.01) 
and non- significant results for pilot type (p>0.05). Subsequent post-hoc analysis produced two subsets, as 
shown in Table 73. For the VFR approach with the BRD display concept, evaluation pilots relied 
completely on out-the-window navigation techniques to perform the approach, which resulted in a mean 
RMS lateral error of 1 ,096 ft. The data for the VFR approach for the BRD display concept exhibits some 
unusual characteristics due to the method of lateral flight path error calculation. 
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Figure 27: VFR Approach Lateral Error for Each Display, Each Pilot Group 
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During the VFR approaches with the BRD display concept, pilots would sometimes cut the comer 
from the downwind leg to the base leg and again from the base leg to final approach, which resulted in a 
tighter approach than desired. If the aircraft became closer to the final approach leg than the downwind 
leg, the lateral flight path error calculation reference would jump from the downwind leg to the final 
approach leg. However, since the aircraft was still flying a track angle closer to the downwind leg than 
final approach, the sign on the error would jump from negative to positive. The primary effect was that 
the magnitude of the resulting flight path error was less than it should have been. For example, if the EP 
departed prematurely from the downwind leg and had a lateral error of 0.8 nm, the recorded lateral error 
would have been approximately 0.6 nm since that point may have been referenced to the final approach 
course. While this does bring into question the accuracy of the lateral error data for this approach with 
the BRD display concept, the data are considered useable for the intent of this report to approximately 
describe the ability of pilots to fly a specific VFR approach path with the BRD display concept. 

For the advanced display concepts, EPs had several other sources of guidance and navigation to work 
with during the VFR approach and were able to achieve much lower levels of lateral flight path error as 
can be seen in Figure 27. The SCFD reduced lateral flight path error to 141.2 ft. The PBG and SVS 
display concepts reduced lateral flight path error further to 70.2 and 79.7 ft, respectively. Note that the 
increased lateral flight path error for the SCFD, compared to the PBG and SVS display concepts, 
although not significant, was likely associated with the lack of turn anticipation and flight path awareness 
for this display concept. 

Table 73: Mean Lateral Flight-Path Error (ft) SNK Results for Display Concept for all Pilots for 

the VFR Approach 


Display 

Concept 

Number of 
Data 
Points 

Subset 

1 

2 

PBG 

18 

70.2 


SVS 

18 

79.7 


SCFD 

19 

141.2 


BRD 

24 


1096.4 

Significance 


0.35 

1.00 


4. 9.2 ILS approach, all pilots 

An ANOVA performed on the lateral flight path error for the main effects of display concept and pilot 
type resulted in highly significant results for display concept (F(3,85)=25.7, p<0.01) and non-significant 
results for pilot type (p>0.05). Subsequent post-hoc analysis produced two subsets, as shown in Table 74. 
In Figure 28, note that the lateral flight path error for the BRD display concept frequently exceeded the 
0. 1 nm FTE criterion, especially for Segment 2 and initial stages of Segment 3 of the approach. (The 
label “RNP 0.1 criterion” is used for illustration to place significance on 0.1 nm FTE. The data are time 
history data and do not imply 95% probability of exceedance or confidence levels.) Part of the reason 
that the BRD display concept exceeded the RNP 0.1 criterion from approximately 7.0 to 8.4 nm was due 
to the low sensitivity of the localizer course deviation indicator (CDI) at this range to touchdown. During 
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Segment 2, pilots were flying a localizer intercept heading with the BRD display concept and had no 
direct indication regarding the location of the desired path which contributed to large lateral flight path 
errors. 

The effect of pilot group can also be seen in Figure 28, but only for the BRD display concept, where 
the VFR pilots exhibited worse lateral flight path error than the IFR or H-IFR pilots groups. No 
discernible difference is observable for pilot type for the other display concepts. One ILS approach that 
required termination can be seen in Figure 28 by the red line that terminates at about 2.5 nm from 
touchdown. For this run, the VFR pilot had no awareness of desired heading or relationship to the 
intended course, as evidenced by a failure to capture and track the ILS resulting in crossing back and forth 
across the final approach flight path. 
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Figure 28: IFR Approach Lateral Error, By Display Condition and Pilot Group 
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The SCFD display concept reduced lateral flight path error approximately 78% to only 140.1 ft 
compared to the BRD display concept, as shown in Figure 28 and Table 74. Flowever, results in Figure 
28 indicate that the SCFD display concept exceeded the 0.1 nm (i.e., FTE) criterion in Segment 2, likely 
the result of the lack of turn anticipation for this display concept. Lateral flight path error results for the 
PBG concept were better than the SCFD, reducing lateral flight path error by 47%. The SVS display 
concept had the lowest lateral error of all and was slightly better than the PBG. Except for a few very 
brief times during the initial stages of the approach, the PBG and SVS display concepts were well within 
the 0. 1 nm FTE criterion for all pilot groups. For lateral flight path error results, it should be noted that 
evaluation pilots were responding to CDI needle deflections for the BRD display concept, and, at large 
distances from the runway, small angular CDI deflections represented large flight path errors. 
Conversely, SCFD, PBG, and SVS display guidance concepts were based on rectilinear flight path errors 
and provided consistent guidance cues throughout the approach. Figure 28 also illustrates that the only 
display concept to indicate a difference between the different pilot groups was the BRD display concept. 

Table 74: Mean Lateral Flight-Path Error (ft) SNK Results for Display Concept for all Pilots for 

the ILS Approach 


Display 

Concept 

Number of 
Data Points 

Subset 

1 

2 

SVS 

21 

72.9 


PBG 

16 

73.8 


SCFD 

18 

140.1 


BRD 

30 


624.1 

Significance 


0.684 

1.000 


4.9.3 VMC-Like approach, all pilots 

Figure 29 presents lateral flight path error results for the VMC-like approach. It can be seen from 
Figure 29 that the SCFD display concept exceeded the RNP 0. 1 criterion, especially during the turn to 
final approach (i.e., at 1.0 nm), but also during the turn to base leg to a lesser degree (i.e., at 2.4 nm). 
Results from Figure 29 also show that all three pilot types performed similarly. An ANOVA performed 
on the lateral flight path error for the main effects of display concept and pilot type resulted in highly 
significant results for display concept (F(3,56)=l 1.9, p<0.01) and non-significant results for pilot type 
(p>0.05). Subsequent post-hoc analysis produced two unique subsets, as shown in Table 75. Unlike 
results for the ILS approach, which indicated no statistical difference between the advanced display 
concepts, results for the VMC-like approach indicate that the SCFD display concept was significantly 
worse than the PBG and SVS display concepts. This result is again attributable to the lack of turn 
anticipation of the SCFD display concept. The inclusion of SVS terrain resulted in an 1 1.6% reduction in 
lateral flight path error compared to the PBG display concept. Lateral flight path error results for the SVS 
display concept were nearly one half of (48.8% less) those for SCFD display concept. 
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Table 75: Mean Lateral Flight-Path Error (ft) SNK Results for Display Concept for all Pilots for 

the VMC-Like Approach 


Display 

Concept 

Number of 
Data Points 

Subset 

1 

2 

svs 

19 

70.9 


PBG 

19 

80.2 


SCFD 

18 


138.6 

Significance 


0.560 

1.000 
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Figure 29: VMC-Like Approach Lateral Error, By Display Concept and Pilot Group 
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Figure 30: Lateral Error for Segment 3 for the ILS and VMC-Like Approach, All Pilots 


4.9.4 ILS approach, Segment 3, without VFR pilot data 

Results for lateral flight path error were analyzed without the VFR pilot data during Segment 3, 
enabling a more direct comparison of the four display concepts within current IMC operations. Results 
from an ANOVA performed on the lateral flight path error results for the main effects of display concept 
and pilot type revealed that display concept was highly significant (F(3,56)=22.92, p<0.01) while pilot 
type was not significant (p>0.05). There were no significant interactions between display concept and 
pilot type. A subsequent SNK post-hoc analysis yielded two unique subsets, as shown in Table 76. The 
lowest (best) subset was composed of all of the advanced display concepts with an average mean of 
approximately 54 ft, which was only one fifth of the lateral flight path error for the BRD display concept. 
This result is consistent with previous research and indicates that large decreases in flight path error are 
possible with advanced display concepts. The highest (worst) subset contained only the BRD display 
concept with a mean of 272 ft, which, for the purposes of this report, establishes the acceptable 
performance for the ILS approach in today’s operations. Also note that the effect of pilot type is only 
apparent for the BRD display concept in Figure 28, indicating the capability of the advanced display 
concepts, especially the PBG and SVS, to greatly enhance the ability of VFR pilots to maintain lateral 
path error as good as higher-hour better-trained pilots, for the ILS approach. 
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Table 76: Mean Lateral Flight-Path Error (ft) SNK results, for the ILS Approach, IFR and H-IFR 

Pilots 


Display Concept 

Number of 
Data Points 

Subset 

1 

2 

PBG 

10 

49.1 


SCFD 

12 

54.5 


SVS 

15 

59.3 


BRD 

19 


272.4* 

Significance 


0.96 

1.00 


*Note: This value is used herein as the baseline performance for an ILS Approach 


4.9.5 ILS and VMC-Like approach analysis, Segment 3, all pilots 

An analysis of the Segment 3 data for the ILS and VMC-like approaches was performed to compare 
the differences between nominal ILS localizer and glideslope tracking to the turning and descending flight 
of the VMC-like approach. Figures 28 and 29 provide a visual comparison of the results for the ILS and 
VMC-like approaches to assess the capabilities of advanced display concepts to support advanced IMC 
operations. Note that results for the SCFD display concept show that the “RNP 0. 1 criterion” (i.e., FTE 
of 0. 1 nm) was frequently exceeded in several runs at approximately 0.9 nm during the turn to final 
approach. There was also one run for an IFR pilot that exceeded the RNP 0. 1 criterion during the turn to 
base leg at approximately 2.2 nm. Results for the PBG and SVS display concepts indicate reduced lateral 
flight path error compared to the SCFD display concept and remained well within the 0.1 nm FTE (“RNP 
0. 1 criterion”). There was also an increase for lateral flight path error for the VMC-like approach 
compared to the ILS approach for Segment 3. This was expected due to the nature of the VMC-like 
approach with two turns during Segment 3. 

An ANOVA was conducted on the lateral flight path error data for all pilots for Segment 3 of the ILS 
and VMC-like approaches. The ANOVA analysis indicated that display concept (F(2,l 1 1)=15.05, 
p<0.01) and approach (F(l , 1 1 1)=28.63, p<0.01) were highly significant for lateral path error while pilot 
type (F(2,l 1 1)=2.15, p=0.12) was not significant. The interactions between display concept and approach 
were also highly significant (F(2,l 1 1)=12.29, p<0.01); however, the interaction between pilot type and 
display concept, and pilot type and approach, were not significant. Subsequent post-hoc analysis yielded 
two subsets as shown in Table 77. In the lower (best) subset were the SVS and PBG display concepts 
with an average mean lateral flight path error of approximately 65 ft, which was approximately one half 
of the error for SCFD. One reason for the reduced lateral flight path error for the SVS and PBG display 
concepts involves the turn anticipation provided by the HITS guidance. Even in the presence of the ND 
that provided route and turn anticipation information to the evaluation pilots, the HITS guidance was 
especially valuable for the VMC-like approach. 
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Table 77: Mean Lateral Flight-Path Error (ft) SNK results, Segment 3 of the ILS and VMC-Like 

Approaches for all Pilots 


Display Concept 

Number of 
Data Points 

Subset 

1 

2 

SVS 

40 

64.2 


PBG 

35 

65.8 


SCFD 

36 


118.0 

Significance 


0.88 

1.000 


Table 78, along with Figures 28, 29, and 30, provides information regarding the effect of approach and 
display concept that illustrates the increase in lateral flight path error for the VMC-like approach 
compared to the ILS approach. Both the SCFD and PBG display concepts exhibited larger increases in 
lateral flight path error than the SVS display concept due to approach in Segment 3. Lateral flight path 
error for the VMC-like approach increased by factors of 2.78, 1.48, and 1.16 for the SCFD, PBG, and 
SVS display concepts, respectively, compared to the ILS approach. The smaller increase in lateral error 
for the VMC-like approach can be attributed to the increased situation awareness provided by the HITS 
guidance and SVS terrain for the VMC-like approach. 

Table 78: Lateral Flight-Path Error Statistical Results (ft), Segment 3 of the ILS and VMC-Like 

Approaches for all Pilots 


Display 

Concept 

Approach 

Mean 

Standard 

Deviation 

SCFD 

ILS 

62.4 

25.7 


VMC-Like 

173.6 

113.7 

PBG 

ILS 

52.2 

18.8 


VMC-Like 

77.3 

30.2 

SVS 

ILS 

59.5 

23.7 


VMC-Like 

69.3 

29.6 


4. 9. 6 Test condition analysis, Segment 3 

An ANOVA analysis was performed on lateral flight path error for TC for Segment 3 for all three 
approaches with post-hoc results presented in Table 79. Test Condition was revealed to be highly 
significant for lateral flight path error results (F(12,220)=34.724, p<0.01) and was not significant for pilot 
type (p>0.05). Two unique subsets were formed in subsequent post-hoc analysis. The highest (worst) 
subset was comprised of TC 1 (VFR approach with BRD display concept). This result was expected in 
that the EPs had to rely exclusively on their out-the- window navigation to negotiate the VFR approach 
and is consistent with current VFR operations. There were no significant lateral path error differences 
among the other test conditions. It is important to note that results for TCs 3, 4, 6, 7, 8, 10, and 1 1 had a 
group average mean of 70 ft, which was only approximately one third of the lateral flight path error 
results for TCs 2, 5, and 9, with an average mean of 206 ft. Although not statistically significant, the TCs 
for the advanced display concepts for the IMC approaches (i.e., TCs 6 through 11) were all lower than for 
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TC 5 (ILS approach for BRD). In addition, results for the VFR pilots’ evaluations of the SVS display 
concept for the ILS and VMC-like approaches were also lower than TC 5. Based on lateral flight path 
error, advanced IMC approaches, such as the VMC-like approach, appear feasible for the PBG and SVS 
advanced displays tested in this study. From Figure 29, it can be seen that results for the SCFD display 
concept violated the RNP 0.1 criterion (i.e., 0.1 nm FTE) for the VMC-like approach, indicating that this 
type of manually- flown approach may not be appropriate for this display concept. 


Table 79: Mean Lateral Flight-Path Error (ft) SNK results for TC 


Test Condition 

TC# 

Number of 
Data Points 

Subset 

1 

2 

ILS Approach + PBG 

7 

16 

52.2 


ILS Approach + SVS 

8 

21 

59.5 


ILS Approach + SVS (VFR pilots) 

12 

6 

59.8 


ILS Approach + SCFD 

6 

18 

62.4 


VMC-Like Approach + SVS 

11 

19 

69.3 


VMC-Like Approach + PBG 

10 

19 

77.3 


VFR Approach + PBG 

3 

18 

77.9 


VMC-Like Approach + SVS (VFR pilots) 

13 

6 

84.2 


VFR Approach + SVS 

4 

18 

89.7 


VFR Approach + SCFD 

2 

19 

172.7 


VMC-Like Approach + SCFD 

9 

18 

173.6 


ILS Approach + BRD (no VFR pilots) 

5 

19 

272.4 


VFR Approach + BRD 

1 

24 


1454.9 

Significance 



0.747 

1.000 
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4.10 Vertical Flight Path Error 

No clear criterion were available for the vertical flight path error for the IMC approaches. For the 
discussions herein, +/-100 ft will be used as a relevant criterion for comparisons and assessments. 
Figures 31, 32, and 33 show real-time vertical flight path error data for the VFR, ILS and VMC-like 
approaches, respectively. Figure 34 and Table 80 show mean RMS results for vertical flight path error. 


Table 80: Vertical Path Error (ft) Descriptive Statistics for all Pilots 


Display 

Concept 

Approach 

Mean 

Standard 

Deviation 

Number of 
Data Points 

BRD 

VFR 

99.6 

46.6 

24 


ILS 

171.8 

117.6 

30 


Total 

139.7 

99.1 

54 

SCFD 

VFR 

25.2 

15.0 

19 


ILS 

25.6 

15.0 

18 


VMC-Like 

26.6 

15.1 

18 


Total 

25.8 

14.8 

55 

PBG 

VFR 

31.7 

12.3 

18 


ILS 

40.7 

19.2 

16 


VMC-Like 

35.2 

16.7 

19 


Total 

35.7 

16.3 

53 

svs 

VFR 

36.9 

22.4 

18 


ILS 

39.7 

18.0 

21 


VMC-Like 

30.1 

13.8 

19 


Total 

35.7 

18.4 

58 

Total 

VFR 

51.9 

43.0 

79 


ILS 

83.5 

96.4 

85 


VMC-Like 

30.7 

15.4 

56 


Total 

58.7 

68.8 

220 


4.10.1 VFR approach 

For the VFR approach, the EPs were instructed to maintain altitude until abeam the runway numbers, 
then descend through base leg and final approach segments. For the BRD display concept, no guidance 
information was provided to the EPs and they relied on the out-the-window view to manage flight path 
along with cross checking their altimeter at specific points along the approach. As can be seen in Figure 
31 and Table 81, vertical flight path error for the BRD display concept was approximately 100 ft, which is 
reasonable for this type of approach. 

Note that issues with the VFR approaches that were flown with cut-comers at the turn to base leg from 
the downwind leg created some problems with the vertical flight path error data as well. This was 
previously described for the lateral flight path error. The main problem was that when the lateral flight 
path error calculation shifted the desired flight path reference point to the final approach leg when the 
comer between the downwind leg and base leg was cut, the desired altitude also changed. Unlike lateral 
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flight path error, this anomaly usually increased the magnitude of the vertical flight path error results for 
the BRD display concept. However, the data does approximately represent the capability of pilots to fly a 
VFR approach with the BRD display concept especially for Segment 2. 

Provisions of guidance cues and vertical flight path error information within the advanced display 
concepts enabled EPs to substantially lower their vertical flight path errors to approximately 40% of the 
results for the BRD display concept, as shown in Figure 31 and Table 81. An ANOVA conducted on the 
vertical flight path error data for the VFR approach for the main effects of display concept and pilot type 
resulted in significant results for display concept (F(3, 79)=45.0, p<0.01) but not for pilot type (p>0.05) or 
the interaction of pilot type with display concept. 

Table 81: Mean Vertical Flight-Path Error (ft) SNK Results for Display Concept for all Pilots for 

the VFR Approach 


Display 

Concept 

Number of 
Data Points 

Subset 

1 

2 

SCFD 

19 

25.2 


PBG 

18 

31.7 


svs 

18 

25.2 


BRD 

24 


99.6 

Significance 


0.313 

1.000 
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Figure 31: VFR Approach, Vertical Error, By Display Concept and Pilot Group 
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4.10.2 ILS approach, all pilots 

An ANOVA conducted on the vertical flight path error data for the ILS approach for the main effects 
of display concept and pilot type produced highly significant results for display concept (F(3, 85)=25.7, 
p<0.01) but not for pilot type (p>0.05). There was no significant interaction between display concept and 
pilot type. Subsequent post-hoc analysis on display concept resulted in two unique subsets, as shown in 
Table 82. For the advanced display concepts, vertical flight path error was reduced approximately by a 
factor of five compared to the BRD display concept, as shown in Figure 32. This result was partially due 
to the SCFD and HITS guidance systems providing constant sensitivity to vertical flight path errors for 
the entire ILS approach and the advanced display concepts integrated information format. Note that one 
dot of vertical GDI deflection corresponds to approximately 360 ft of vertical flight path error at the 
glideslope intercept point. There is also an indication that the SCFD display concept had lower vertical 
flight path error than the PBG and SVS display concepts which had pathway-based guidance as indicated 
in Figure 32. The reduced vertical flight path error results for the SCFD was due to the combination of 
the prominent single-cue-flight-director symbology with the pitch-attitude command system that 
emphasized attitude and vertical error control. While the effect of pilot type was not statistically 
significant, the operational effect of pilot type can be observed in Figure 32 where the VFR pilots’ results 
can be seen to be somewhat worse than the other two pilot groups for the BRD and PBG display 
concepts. The effect of pilot group was reduced for the SVS display concept as seen in Figure 32, which 
appears to show the positive effect of SVS terrain to aid in vertical error reductions when compared to 
results for the PBG display concept. The presence of the SVS terrain significantly increased pilots’ 
situation awareness and their ability to integrate the information provided, likely leading to reduced 
vertical flight path error. 

Table 82: Mean Vertical Flight Path Error (ft) SNK Results for Display Concept for all Pilots, ILS 

Approach 


Display 

Concept 

Number of 
Data Points 

Subset 

1 

2 

SCFD 

18 

25.6 


SVS 

21 

39.7 


PBG 

16 

40.7 


BRD 

30 


171.8 

Significance 


0.761 

1.000 
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Vertical Error, ft 




Figure 32: ILS Approach, Vertical Error, By Display Concept and Pilot Group 


119 


4.10.3 VMC-Like approach, all pilots 

Results for vertical flight path error, illustrated in Figure 33, indicate somewhat similar 
characteristics for all display concepts for the VMC-like approach. Flowever, the PBG display concept 
does show some degraded performance compared to the SCFD, with several vertical error excursions in 
excess of +/-100 ft. An ANOVA conducted on the vertical flight path error data for the VMC-like 
approach for the main effects of display concept and pilot type did not produce any significant (p>0.5) 
results for display concept, pilot type, or interaction between these two factors. 
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Figure 33: VMC-Like Approach, Vertical Error, By Display Concept and Pilot Group 
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4.10.4 ILS approach, Segment 3, without VFR pilot data 

Vertical flight path error was analyzed without the VFR pilot data. Results from an ANOVA 
revealed that display concept was highly significant (F(3,56)=19.634, p<0.01) for vertical path error. 
Neither pilot type (i.e., either IFR or H-IFR) nor the interaction between pilot type and display concept 
were significant (p>0.05) for this measure. These results match the effects shown in Figure 32. 
Subsequent post-hoc analysis yielded two unique subsets, as shown in Table 83. The advanced display 
concepts comprised the lower (best) vertical path error subset with the BRD display concept being the 
only member of the higher (worst) subset. The lowest vertical flight path error was for the SCFD display 
concept with results being approximately one half as high as the results for the PBG and SVS display 
concepts. The differences between the SCFD, PBG, and SVS display concepts were not statistically 
significant, but this result may indicate the ability of the SCFD display concept to improve vertical flight 
path error for this approach segment. The highest (worst) subset contained only the BRD display concept 
with a mean error of 145 ft, which established the nominal performance for the ILS approach in today’s 
operations for the data considered in this report. Overall, the average of the mean vertical flight path error 
for the advanced display concepts was 27.2 ft, which was less than one-fifth of the error for the BRD 
display concept. Again, this result is likely due to the variable sensitivity of the CDI needle movement in 
response to vertical flight path error with range during the ILS approach and the integrated, intuitive 
guidance provided by the SCFD and HITS guidance systems. These results suggest that terrain clearance 
margins required for ILS operations today could be greatly reduced with advanced display concepts. 
Currently, approach procedures are designed according to Reference 24 (FAA Order 8260.3, United 
States Standard for Terminal Instrument Procedures (TERPS)). For a nominal 3-degree glideslope, the 
altitude clearance is approximately 892 ft at the beginning of Segment 3 (i.e., 8.4 nm from touchdown). 

If it is assumed that total system error is composed of navigation system error and flight technical error, 
then it is possible that this amount of margin could be reduced if flight technical error is significantly 
reduced. 

Table 83: Mean Vertical Flight-Path Error (ft) SNK results for the ILS Approach, IFR and H-IFR 

Pilots Only 


Display 

Concept 

Number of 
Data Points 

Subset 

1 

2 

SCFD 

12 

16.2 


PBG 

10 

32.5 


SVS 

15 

33.0 


BRD 

19 


145.0* 

Significance 


0.70 

1.0 


*Note: This value is used herein as the baseline performance for an ILS Approach 


4.10.5 ILS and VMC-Like approach analysis, Segment 3, all pilots 

Figures 32 and 33 show a visual comparison of the vertical flight path errors for the advanced display 
concepts for the ILS and VMC-like approaches. An ANOVA was conducted on the vertical flight path 
error data for all pilots for Segment 3 of the ILS and VMC-like approaches for the main effects of pilot 
type, display concept, and approach to compare and contrast each advanced display concept as well as 
assess the capabilities of advanced display concepts to support advanced IMC operations. The ANOVA 
analysis indicated that pilot type (F(2,l 1 1)=7.701, p<0.01) and display concept (F(2,l 1 1)=8.96, p<0.001) 
were highly significant and approach was not (p>0.05) for vertical path error. There were no significant 
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interactions between the main effects for this measure. Table 84 shows that the IFR and H-IFR pilot 
groups exhibited significantly less vertical flight path error than the VFR pilot group. The VFR pilots’ 
vertical flight path error was approximately 10 ft, or 37%, more than the IFR and H-IFR pilots. Table 84 
provides SNK results for display concept for the ILS and VMC-lIke approaches and shows that the SCFD 
had significantly less vertical path error (approximately 12 ft) than the PBG and SVS display concepts. 
There were no significant vertical path error differences between the PBG and SVS display concepts. 

This result is likely due to the superior vertical flight path guidance provided by the SCFD system for 
these approaches. The SCFD symbology provided a compelling indication of the commanded pitch 
attitude to control vertical flight path error. However, as shown previously, the SCFD did exhibit poor 
lateral flight path error results primarily due to the lack of turn anticipation. Overall, vertical flight path 
error results indicate that the VMC-like approach may be feasible for the advanced display concepts. 

Table 84: Mean Vertical Flight-Path Error (ft) SNK results for Pilot Type, ILS and VMC-Like 

Approaches for all Pilots 


Pilot Type 

Number of 
Data Points 

Subset 

1 

2 

IFR 

36 

27.6 


H-IFR 

39 

28.6 


VFR 

36 


39.1 

Significance 


0.765 

1.0 


Table 85: Mean Vertical Flight-Path Error (ft) SNK results for the Advanced Display Concepts, 

ILS and VMC-Like Approaches for all Pilots 


Display 

Concept 

Number of 
Data Points 

Subset 

1 

2 

SCFD 

36 

23.6 


SVS 

40 


33.9 

PBG 

35 


37.6 

Significance 


1.000 

0.252 


4.10.6 Test condition analysis, Segment 3 

A subsequent analysis was performed on vertical flight path error based on TC, with post-hoc results 
presented in Table 86. TC was highly significant for vertical flight path error results (F(12,209)=20.555, 
p<0.01). Three unique subsets were formed from SNK post-hoc analysis of the vertical flight path error. 
The lowest vertical path error was recorded for the advanced display concepts regardless of approach 
(VMC-like, ILS, or VFR). The VFR approach evaluation of the BRD display concept had significantly 
worse vertical path error than any of the advanced displays for any approaches, but had significantly 
better vertical-path error than TC 5 (ILS approach with BRD display concept without the VFR pilots). 
The BRD display concept for the ILS approach had significantly worse vertical path error than all other 
display concept and approach combinations. These results appear to indicate that advanced IMC 
approaches similar to the VMC-like approach, are feasible with the advanced display concepts from the 
standpoint of vertical flight path error, since all IMC evaluations of these display concepts (i.e., TCs 9, 
10, 1 1) were significantly lower than the results for ILS approach with the BRD display concept (TC 5). 
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Results for the VFR pilots’ evaluation of the SVS display concept for both the 1LS and VMC-like 
approaches were significantly better than TC 5. 


Table 86: Mean Vertical Flight-Path Error (ft) SNK results for TC 


Test Condition 

TC 

# 

Number of 
Data Points 

Subset 

1 

2 

3 

ILS Approach + SCFD 

6 

18 

21.7 



VMC-Like Approach + SCFD 

9 

18 

25.5 



VFR Approach + SCFD 

2 

19 

25.9 



VMC-Like Approach + SVS 

11 

19 

29.7 



VMC-Like Approach + SVS (VFR pilots) 

13 

6 

30.9 



VFR Approach + PBG 

3 

18 

31.1 



VMC-Like Approach + PBG 

10 

19 

36.0 



ILS Approach + SVS 

8 

21 

37.6 



ILS Approach + PBG 

7 

16 

39.4 



VFR Approach + SVS 

4 

18 

40.2 



ILS Approach + SVS (VFR pilots) 

12 

6 

49.0 



VFR Approach + BRD 

1 

24 


116.7 


ILS Approach + BRD (no VFR pilots) 

5 

19 



145.0 

Significance 



0.704 

1.000 

1.000 
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Figure 34: Vertical Error, Segment 3, for Approaches 2 and 3, All Pilots 
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5.0 Summary of Results 

Qualitative results for the SVS display concept evaluations in simulated IMC were not statistically 
different from the VMC evaluations, but were statistically different from the IMC evaluations of the BRD 
display concept. This was true for all qualitative parameters used for this study (i.e., SART, TLX, DRR, 
and TA). Note that the reference condition for the IMC evaluation of the BRD did not include the VFR- 
rated pilots’ data to provide a better comparison metric that had strong ties to existing operations. The 
incorporation of SVS terrain provided a significant improvement in the results compared to the PBG 
display concept for situation awareness measures (SART and TA). The effect of the SVS terrain 
depiction decreased workload measures (TLX, DRR) compared to the PBG display concept, but not 
significantly. 

Quantitative results, such as PTS, lateral, vertical, localizer and glideslope errors, indicate that the 
advanced display concepts (SCFD, PBG, and SVS) provided significantly better performance than the 
BRD display concept for the ILS approach. Lateral and vertical error results for the advanced displays 
were reduced by approximately a factor of 4 for the ILS approach compared to the BRD display concept. 
Significant effects of pilot type were observed in several parameters, such as PTS, localizer, glideslope, 
and IAS errors; however, those effects were primarily limited to results from the BRD display concept, 
with the VFR-rated pilots performing significantly worse than the IFR and H-IFR pilots. The SCFD 
display concept generated significantly worse lateral flight path and localizer error results compared to the 
PBG and SVS display concept evaluations for the VMC-like approach. This is thought to be due to a lack 
of turn anticipation of the SCFD. While RMS values of lateral flight path and localizer errors were not 
statistically different from those from the BRD ILS evaluations by the IFR and H-IFR pilots, graphical 
results indicated that PTS and FTE standards were exceeded for the SCFD display concept during the 
turns to base leg and final approach for the VMC-like approach. Several exceedances of the IFR PTS 
criteria were also observed for the PBG display concept for the VMC-like approach during the last 2 nm. 
This indicates that the SCFD and PBG display concepts would not likely be suitable for advanced IMC 
approaches, such as the VMC-like approach. Results for the SVS display concept indicate that all pilots 
can fly an advanced VMC-like approach and generate flight technical error (i.e., lateral, vertical, localizer, 
glideslope and IAS errors) results as good as or better than IFR and H-IFR pilots using the BRD display 
concept for the ILS approach. Glideslope error results for the SCFD display concept were significantly 
superior to the PBG display concept results due to the emphasis on attitude control and the underlying 
flight director guidance algorithm performance. However, pilot workload was increased significantly for 
the SCFD compared to the PBG display concept. The incorporation of SVS terrain for the SVS display 
concept improved flight path control and significantly reduced glideslope error compared to the PBG 
display concept while maintaining workload. Glideslope error results for the SVS display concept were 
similar to results for the SCFD. The depiction of terrain on the SVS display concept greatly improved the 
performance of VFR-rated pilots. 


6.0 Limitations of Current Study 

The current investigation evaluated scenarios in which all systems were functioning nominally over a 
small, but critical, range of conditions and with a limited, but representative, sample of the pilot 
population. Results from this flight test are intended to be applicable to other real-world conditions; 
however, several limitations of the current study dictate that subsequent applications of these results need 
to be performed carefully. 
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Issues related to off-nominal system performance were not addressed, such as ADAHRS failures, 
display failures, and communication failures. One particular off-nominal system issue with SVS display 
technology not addressed in this report was the risk of terrain database errors. Terrain database errors 
could influence a pilot to deviate from the intended flight path due to false SVS terrain cues. It needs to 
be realized that the concept of operations considered herein still involves a visual segment of the approach 
and landing, similar to current ILS approaches. Visually verifying the condition of the runway prior to 
landing will still be performed by the pilot, whether they are using SVS or not, under conceivable GA 
operations in the foreseeable future. 

Potentially increased heads-down time during marginal VMC operations is not explicitly addressed 
within the current study, although some discussion is provided regarding the presence of terrain heads-up 
and heads-down with SVS displays for VMC operations. While a safety of flight issue, VMC mid-air 
collisions are less prevalent than IMC Controlled Flight Into Terrain (CFIT) accidents. As a result, while 
operations in VMC flight may need to be evaluated with SVS displays, it is proposed that significant 
improvements in IMC flight could potentially more than offset this issue and improve the overall safety 
situation. While not part of this research effort, careful concept of operations development and training 
would be needed to effectively deal with issues such as overconfidence and decision making with SVS 
displays. The current work documented herein does reflect the potential benefits from SVS display 
concepts, and other advanced display concepts, and does support the need for further study as well as 
operational evaluations. 

A total of 19 pilots participated overall in this flight test, generating 219 research evaluations. More 
evaluation pilots would have helped to add to the statistical power of the subsequent analyses, especially 
those that compared across pilot groups. The use of pilot hours and officially logged training as a control 
variable to establish pilot type could be improved. For example, some low-time VFR pilots 
acknowledged that they frequently used desktop simulation tools to practice IFR approaches. This type of 
auxiliary experience could be mitigated through pre-testing of subject pilots (i.e., to see that they provide 
consistent performance for their group) and also pre-test screening (to screen-out pilots with known 
discrepancies). 

Another more basic limitation of the study was the ability of current HF tools (e.g., SART, TLX, 
DRR) that were employed to accurately differentiate the effects of all of the factors involved. In general, 
these HF parameters are used to compare human performance using one display or operational concept to 
another display or operational concept. Results are usually limited to saying that one display, operational 
concept, etc. is as good, as or better than, another display or operational concept. There is some concern 
about using these metrics for comparisons across largely different scenarios, such as encountered between 
VMC and IMC flight. 

7.0 Conclusions 

A flight test was performed to compare the use of three advanced primary flight and navigation 
display concepts to a baseline, round-dial concept to assess the potential for advanced operations. The 
displays were evaluated during visual and instrument approach procedures including an advanced 
instrument approach resembling a visual airport traffic pattern. Nineteen pilots from three pilot groups, 
reflecting the diverse piloting skills of the General Aviation pilot population, served as evaluation 
subjects. Specifically, three hypotheses characterizing potential transformations were defined. The 
hypotheses can be summarized as follows: 

1. Using SVS displays, the workload and SA results of all pilots flying landing approaches 
in IMC will be as good as, or better than, the same pilots flying in VMC using baseline 
round-dial displays. 
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2. Using SVS displays, low-time, non-instrument-rated pilots flying nominal ILS approaches 
in IMC will achieve flight technical error, workload, and SA results as good as, or better 
than, ratings from instrument-rated pilots conducting the same ILS approach using 
baseline, round-dial displays. 

3. Using SVS displays, all pilots, including low-time, non-instrument-rated pilots, will be 
able to conduct IMC approaches with geometries similar to a VFR pattern with flight 
technical error, workload, and SA results as good as, or better than, results from 
instrument rated pilots conducting ILS approaches using baseline round-dial displays. 

The flight test had four unique elements to assess these hypotheses. The first was including non- 
instrument-rated pilots in the evaluation of instrument procedures in simulated IMC. This inclusion 
allowed direct comparison of the performance and ratings of non-IFR-rated pilots using SVS displays and 
IFR-rated pilots using conventional instrumentation. The second element was including operations in 
both VMC and simulated IMC. This aspect of the test enabled direct comparisons of workload and 
situation awareness ratings across these very different operational environments. The third element was 
the inclusion of an instrument approach pattern that overlaid a typical VFR airport traffic pattern. 
Including this element allowed direct comparison of this advanced operation conducted with SVS display 
versus conventional operations and displays. The fourth element was including a progression of 
advanced display concepts. This aids in assessing display elements that are essential towards enabling 
significant operational changes. 

Relative to the three hypotheses, the results of the test can be summarized as follows: 

Hypothesis 1 is supported by the analyses for SART (situation awareness) and TLX (pilot workload). 
Results for the SVS display concept evaluations for the ILS and VMC-like approaches were significantly 
better than the SCFD and PBG display concepts for these measures. SART and TLX results for the SVS 
display concept IMC evaluations were as good as or better than the VFR evaluation of the BRD display 
concept. These results show that using as SVS display, the workload and SART ratings of all the subject 
pilots flying landing approaches in IMC were as good as, or better than, the same pilots flying in VMC 
using baseline round-dial displays. 

Hypothesis 2 is supported by analysis that revealed that the VFR pilots’ results for the SVS display 
concept ILS evaluations were improved for all of the flight technical error variables considered (PTS, 
lateral error, localizer error, and glideslope error), compared to those from the IFR and H-IFR pilots for 
the ILS evaluation of the BRD display concept. Also, VFR-pilot results for the SVS display concept 
during the ILS evaluations were significantly improved for vertical error as well as SART and TLX 
results. While a bias in the VFR pilots’ results was encountered that illustrated a tendency of these pilots 
to provide higher SART ratings than the other two pilot groups, the increase in SA provided by SVS 
displays is considered much larger than the bias and more than compensated for this artifact in the data. 
IAS results indicate that VFR pilots’ performance was slightly worse for the SVS display concept 
compared to the IFR and H-IFR pilots’ evaluation of the BRD for the ILS approach, but not significantly. 
Regarding Hypothesis #2, these results show that using an SVS display, low-time, non- instrument-rated 
pilots flying nominal ILS approaches in IMC can achieve flight technical error, workload and SART 
ratings as good as, or better than, ratings from instrument-rated pilots conducting the same ILS approach 
using baseline round-dial displays. 

Hypothesis 3 is supported by all pilot types, including low-time VFR-rated pilots, generating similar 
or much better flight-technical- and IAS-error results for the SVS display concept evaluations for the 
VMC-like approach compared to the IFR and H-IFR pilots using the BRD display concept for the ILS 
approach. Analysis also revealed that results for the VMC-like approach using the SVS display concept, 
evaluations were improved, but not significantly, for many of the variables considered (PTS, lateral error, 
localizer error, and glideslope error). Vertical error, as well as SART and TLX results, were significantly 
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improved for the SVS display for the VMC-like approach compared to the IFR and H-IFR pilot 
evaluations of the BRD display concept for the ILS approach. The IAS results indicate that performance 
was slightly worse for the SVS display concept for the VMC-like approach compared to the IFR and H- 
IFR pilots’ evaluation of the BRD for the ILS approach, but again not significantly. With regard to 
hypothesis #3, these results show that using an SVS display, all the subject pilot types, including low- 
time, non-instrument rated pilots, were able to conduct IMC approaches with geometries similar to a VFR 
pattern with flight technical error, with workload and SART ratings as good as, or better than ratings from 
instrument-rated pilots conducting ILS approaches using the baseline round-dial display. 

Considering all 3 hypotheses, all advanced display concepts were able to satisfy hypothesis #2 as a 
result of the VFR pilots being able to generate flight technical error results as good as or better than the 
IFR and H-IFR pilots’ evaluation of the BRD display concept for the ILS approach. For the VMC-like 
approach only the SVS display concept was able to satisfy hypothesis #3 due to the poor lateral and 
localizer error results for the SCFD, and poor glideslope and vertical error for the PBG. Due to the SVS 
displays’ significantly improved SART results, the SVS display was able to satisfy hypothesis #1 and was 
the only one to be able to satisfy all three hypotheses simultaneously. 

Several other noteworthy observations can be made from the results. While improvements due to the 
moving map ND and integrated PFD were observed compared to the baseline round-dial display concept, 
the results indicate additional variability among the advanced guidance concepts (i.e., SCFD and PBG). 

As a result of the improved flight path awareness, the HITS guidance provided improved results for 
qualitative measures, such as SART and TLX, and also quantitative measures, such as lateral flight path 
error and localizer error. While the SCFD display concept did provide significantly reduce vertical flight 
path and glideslope error compared to the PBG display concept, overall, the improvements in lateral flight 
path and localizer error provided by the HITS guidance provided superior performance enabling more 
potential applications for future RNP based procedures with curved segments. In addition to the expected 
improvements in situation awareness (including terrain) with SVS terrain displayed, improvements in 
glideslope error were also evident with SVS display concept for the VMC-like approach. This suggests 
that the terrain background improves the ability of pilots (in particular VFR pilots) to perceive and control 
glideslope error as compared to a HITS without terrain. This effect was particularly apparent during 
dynamic approach segments such as the turn to final of the VMC-like approach. Pilots were able to use 
the SVS terrain to better monitor attitude, turn rate, and flight path angle during the turns. On final 
approach the presence of the runway model complemented use of the velocity vector and HITS guidance. 

Support for the hypotheses considered in this test suggests that SVS does have a significant effect on 
flight in IMC and may enable new types of operations and significantly alter the training needed to safely 
and comfortably operate in IMC. At the same time, it is important to recognize that this test only 
examined nominal approaches and did not include traffic separation, hazardous weather, equipment 
failures, etc. Also, other important rating factors, such as measures of pilots’ decision-making ability or 
possible overconfidence were not considered. It is recommended that future studies build on this study by 
investigating the transformative proprieties of SVS displays in the presence of more comprehensive 
operational considerations and while stressing the importance of effective decision making by pilots of all 
experience levels. 
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Appendix A: Single-Cue Flight Director Description 

The intent of the single-cue flight director (SCFD) system was to provide the general functionality of 
this type of flight guidance within the data set. In general, flight directors of this type provide pitch and 
roll attitude commands for the pilot to follow to achieve acceptable flight path errors. Figure A1 below 
shows what the Primary Flight Display (PFD) looked like for the SCFD display concept. The pilot’s task 
was to pitch and roll the aircraft to get the orange triangle in Figure A- 1 to just line-up with the yellow 
triangles. The yellow triangles would move up and down to indicate changes in pitch attitude, and rotate 
left or right to indicate commands in roll attitude. Rate of error information, such as the difference 
between the desired and actual flight path angle, is blended with instantaneous error, such as vertical 
flight path error, to provide pitch attitude commands to follow a specific trajectory. This particular image 
is indicating that the pilot needs to roll the aircraft to the right to eliminate lateral and/or angular errors. 

The SCFD system gains were adjusted during flight-test checkout to achieve what was qualitatively 
considered to be reasonable performance for this type of guidance system. Gains adjusted included the 
Offset, Track, Alt, and gamma parameters. For the flight test checkout, several pilots evaluated the 
system performance and provided inputs regarding gain adjustment. The final values of these parameters 
are provided in Table A-l. 



Figure A- 1: Single Cue Flight Director 


Figure A-2 provides the block diagrams used for the roll and pitch attitude commands of the SCFD. 
As can be seen, linear lateral error was combined with angular error to form the roll attitude co mm and. 
The bank angle required to maintain the desired flight path (from the flight plan information) was then 
added in to this signal and compared to the current ha nk angle. This result was then scaled and used to 
rotate the yellow pitch/roll command triangles with respect to the orange reference triangle. 
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Figure A- 2: Single-Cue Flight Director pitch and roll attitude command block diagram. 


Similarly, vertical flight path and flight path-angle errors were combined in the pitch attitude 
command system and added to the normal acceleration required to maintain a coordinated turn with the 
current pitch and roll attitudes. This signal was then compared to the current normal acceleration with the 
difference being scaled and used to the drive the yellow pitch/roll command triangles up/down with 
respect to the orange reference triangle. The Offset, Track, Altitude and Gamma gain parameters were 
tuned to provide acceptable dynamic responses to linear (e.g., lateral and vertical) and angular (e.g., 
ground-track and flight path-angle) errors. 

Table A-l provides definitions of the various components of the SCFD system along with the actual 
values for the Offset, Track, Altitude and Gamma gains used for this flight test. These gains were 
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evaluated in flight during checkout operations and adjusted accordingly to provide reasonable qualitative 
performance of the SCFD system. No specific adjustments were made to this system to compare with the 
Highway-In-The-Sky guidance system used for the Pathway-Based Guidance and Synthetic Vision 
System display concepts. 


Table A-l. Flight Director Parameters 


Symbol 

Type 

Description 

Unit/ 

Value 

Lateral error 

Input 

Ownship lateral distance closest to the flight path 

ft 

Path track 

Input 

Flight plan track angle 

deg 

Track angle 

Input 

Actual ownship track angle 

deg 

Turning roll 

Input 

Desired roll angle (for curved path only) 

deg 

V total 

Var 

True airspeed 

ft/sec 

G 

Const 

Gravitational acceleration 

32.2 ft/sec 2 

Turn radius 

Var 

Turning radius from the flight path 

Ft 

Roll angle 

Input 

Ownship actual roll angle 

Deg 

Offset gain 

Const 

Lateral error gain 

0.033 deg/ft 

Track gain 

Const 

Track error gain 

2.0 

vertical error 

Input 

Current Ownship vertical flight path error 

ft 

path gamma 

Input 

Commanded vertical flight path angle 

deg 

Gamma 

Input 

Actual vertical flight path angle 

deg 


Input 

Ownship roll angle 

rad 

0 

Input 

Ownship pitch angle 

rad 

Nz coord 

Input 

Normal acceleration of a cordinated turn = 
COS0 / COSl)) 

g 

Nz 

Input 

Actual normal acceleration 

g 

Alt gain 

Const 

vertical error gain 

0.068deg/ft 

gamma gain 

Const 

Vertical flight path angle gain 

0.3 g/deg 

T 

Const 

Pitch/roll flight director filter time constant 

sec 


134 




Appendix B: Highway-In-The-Sky Guidance System 

The intent of the Highway-In-The-Sky (HITS) guidance system was to provide the general 
functionality of this type of flight guidance within the data set. In general, flight path guidance systems of 
this type provide lateral and vertical cues for the pilot to follow to achieve acceptable flight path errors. 
Figure B1 below shows what the Primary Flight Display (PFD) looked like for the Synthetic Vision 
System (SVS) display concept. 

For the current study, a concept known as the Crows-Foot tunnel with tadpole guidance was selected. 
This concept provides good situation awareness of the intended flight path without cluttering the display 
and compromising the value of the SVS terrain and is described further in References 3, 4, 8, 9, 1 1 and 
12. The right-angle magenta triads formed the comer points of the desired flight path tunnel. The tunnel 
was set to 2 dots of localizer error and 1 dot of glideslope error and limited to 600 ft wide and 350 ft tall. 

The pilot’s task was to adjust the aircraft’s flight path and track angles to overlay the velocity vector 
(white circle with left, right, and top white fins) with the magenta guidance cue (tadpole). To provide 
enhanced flight path control, the velocity vector was quickened with pitch rate to eliminate the non- 
minimum phase response of the C206 to elevator commands. The tadpole guidance cue was located 
using modified pursuit guidance where the cue was located at a point along the desired flight path 
approximately 15 seconds in front of the aircraft’s current location as described in Reference Bl. The top 
of the magenta guidance cue (also known as a tadpole) would lean in the direction of the upcoming turn 
to provide additional turn awareness. This particular image is indicating that the pilot needs to roll the 
aircraft to the right to eliminate a lateral error. 

Appendix B references: 

Bl : Merrick, V.K., & Jeske, J.A., “Flight Path Synthesis and HUD Scaling for V/STOL Terminal 

Area Operations,” NASA TM-1 10348, April 1995. 



Figure B- 1: SVS Primary Flight Display showing HITS guidance symbology. 
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Appendix C: Questionnaires 


C.l General 

The questionnaires administered in support of this test are included in this appendix. They are listed 
as either post-run or post-test. Post-run questionnaires were those administered immediately after each 
run. Evaluation pilots were provided as much time as they needed to answer the basic block of post-run 
questions. The extended post-run questions were answered whenever time permitted, such as during 
initializations of the ILS approach since it took extra time to perform this operation, or during ATC- 
induced delays. Post -test questionnaires were those administered following a short break after all testing 
was complete. 

C.2 Evaluation Pilot Questionnaire (EPQ) Tool 

In order to expedite the acquisition of qualitative pilot rating data during the flight test, a hand-held 
pocket personal computer was used. Evaluation pilots were instructed to read the question aloud and 
then move the response indicator marker using the stylus to the selected response location. Evaluation 
pilots were also asked to provide any additional verbal comments before selecting the next page. Figure 
C-l shows what the EPQ looked like to the EP for the performance question of the TLX questionnaire 
section. The response scale was similar for most questions with the end descriptors adjusted accordingly 
(i.e., good/poor or high/low etc.). For the DRR question block, the EPs responded yes/no to a series of 
questions and ultimately selected a single numerical score. 



Figure C- 1: Example image of the Evaluation Pilot Questionnaire tool. 
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For the DRR questions, EPs first responded to a series of yes/no questions to navigate through the 
decision tree logic. They then selected the most appropriate response from a limited subset of DRRs. For 
example, if EPs answered yes to “Is the Display Readable?” and yes to “Is adequate performance 
attainable with a tolerable workload” and yes to “Is it satisfactory without improvement?”, EPs could then 
select from DRRs of 1 , 2, or 3 with the associated Characteristics of Display Parameters and Demands on 
Pilot During Scenario phrases. 

C.3 Post-Run Questionnaires, basic block: 

Instructions: Answer the following questions by moving the slider icon to the appropriate location. 
Press the “Next Question” button when finished. 


Situation Awareness Rating Technique (SART) 

SART 1: Demand on Attentional Resources: Rate your overall impression of the scenario in terms 
of how much attention and effort was required to successfully perform the task of piloting the aircraft. 
Items to consider include the likelihood of the situation to change suddenly (instability), the degree of 
complication associated with the scenario ( complexity), and the number of variables changing during the 
scenario (variability). 

SART 2: Supply of Attentional Resources: Rate your overall impression of the scenario in terms 
of how much spare attention was available for secondary tasks after the primary task of piloting the 
aircraft is accounted for. Items to consider include your degree of readiness for the primary task 
(arousal), your level of concentration during the primary task (concentration), the necessity and frequency 
of attention division between the primary task and any sub-tasks (division), and the amount of attention 
left over from the primary task to allocate to new variables of sub-tasks (spare capacity). 

SART 3: Understanding of the Situation: Rate your overall understanding of what was occurring 
during the scenario. Items to consider are the amount of information received and understood during the 
scenario (quantity), the value of the information (quality), and the familiarity you may have had with what 
was taking place during the scenario. 

The SART result is calculated by: SART 2 + SART 3 - SART 1 


Terrain Awareness Question 

Terrain Awareness: Rate your overall understanding of the terrain environment you were operating 
within. Items to consider are the quantity and quality of the terrain information available to you and how 
comfortable you were with your terrain awareness during this scenario. 


NASA Task Load Index (TLX) Workload Measurement Questionnaire 

Mental Demand: How much mental and perceptual activity was required (e.g. thinking, deciding, 
calculating, remembering, looking, searching, etc.)? Was the task easy or demanding, simple or complex, 
exacting or forgiving? 

Physical Demand: How much physical activity was required (e.g. pushing, pulling, turning, 
controlling, activating, etc.)? Was the task easy or demanding, slow or brisk, slack or strenuous, restful or 
laborious? 

Temporal Demand: How much time pressure did you feel due to the rate or pace at which the tasks 
or task elements occurred? Was the pace slow and leisurely or rapid and frantic? 
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Performance: How successful do you think you were in accomplishing the goals of the task set by 
the experimenter (or yourself)? How satisfied were you with your performance in accomplishing these 
goals? 

Effort: How hard did you have to work mentally and physically to accomplish your level of 
performance? 

Frustration Level: How insecure, discouraged, irritated, stressed, and annoyed versus secure, 
gratified, content, relaxed, and complacent did you feel during the task? 

The TLX result is calculated by: (Mental Demand + Physical Demand + Temporal Demand + 
Performance + Effort + Frustration Level)/6.0. Since the Performance scale was oriented in the same 
direction as the other scales, that parameter was subtracted from the summation of the other parameters. 


Hayworth-Newman Display Readability Rating (DRR) Scale: 

The Hayworth-Newman Display Readability Rating scale is shown below is Figure C2. This rating 
scale is also referred to as a Modified Cooper-Harper rating in some literature. Evaluation pilots 
responded to the DRR questions in a decision tree format starting with the first question “Is the display 
readable?”. The DRR scale can be described as having three levels of response. Fevel 1 corresponds to 
responses 1 , 2, or 3 and requires the EP to answer “yes” to the question of “Is it satisfactory without 
improvement?”. Level 2 corresponds to responses 4, 5, or 6 and requires the EP to answer “yes” to the 
question “Is adequate performance attainable with a tolerable workload?” but answer “no” to “Is it 
satisfactory without improvement?”. Level 3 corresponds to 7, 8, 9. A DRR of 10 exceeds Level 3. 
Evaluation Pilots were able to select DRR scores from Level 1, Level 2, Level 3, or DRR=10 depending 
on their responses to the initial decision tree questions. 

Post-run questions, extended block: 

GEOGRAPHICAL SA - Provide an estimate of your geographical situation awareness while flying 
the approach. Items to consider when marking your response include: knowledge of the location of your 
aircraft; other aircraft; terrain features; airports; cities; waypoints; position relative to obstacles; path 
conflicts; and climb/descent points. 

SPATIAL SA - Provide an estimate of your spatial situation awareness while flying the approach. 
Items to consider when marking your response include: knowledge of your aircraft's attitude; altitude; 
heading; velocity; vertical speed; and desired flight path. 

TACTICAL SA - Provide an estimate of your tactical situation awareness while flying the approach. 
Items to consider when marking your response include: identification; location; and flight dynamics of 
other aircraft; own aircraft capabilities in relation to other nearby aircraft; and ATC communications. 

PROJECTED SA - Provide an estimate of your ability to accurately predict the future state of your 
aircraft and the states of nearby aircraft while flying the approach. Items to consider when marking your 
response include: the projected status and dynamics of your aircraft; projected status and dynamics of 
other nearby aircraft; anticipation of possible future conflicts with traffic or obstacles; and threat 
prioritization. 

MENTAL WORKLOAD - Provide an estimate of the amount of mental workload required to 
successfully complete the assigned task of flying the approach. 

PHYSICAL WORKLOAD - Provide an estimate of the amount of physical workload required to 
successfully complete the assigned task of flying the approach. 
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SPARE CAPACITY - Provide an estimate of the amount of mental resources remaining that could 
be used to process new information after all the necessary mental tasks associated with successfully flying 
the approach have been accounted for. 

STRESS - Provide an estimate of the amount of stress (physical and mental) you experienced while 
completing the assigned task of flying the approach. 

DATA LOCATION - The location of flight data on the displays critical to the task of flying the 
approach was efficiently organized minimizing the effort required to gather information. 

DATA VALUE - The flight data presented on the displays was complete, valuable, and had a 
significant impact on my ability to successfully fly the approach. 

DATA CLARITY - The flight data presented on the displays while flying the approach was clear, 
simple to read, and easy to inteipret. 

DATA INTEGRATION - The flight data gathered from the displays was easily and rapidly 
integrated into a meaningful mental representation of the current and future operating status of my 
aircraft. 

DATA CONTENT - The flight data presented on the displays was excessive and included 
unnecessary information 
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Figure C- 2: Hayworth-Newman Modified Cooper Harper (Display Readability). 
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Appendix D: Individual Subjective Data Plots 

This appendix provides individual responses to the pilot questionnaires described in Appendix C and 
reported in the main body of the report. The data presented in this appendix include situation awareness 
(SART), subjective workload (TLX), display readability rating (DRR), and terrain awareness (TA). The 
plots are arranged by approach (i.e., VFR, IFR, and VMC-like) with four plots per parameter for a total of 
12 plots. In the main body of the report, results and conclusions regarding the effects of display concept 
and approach are emphasized. The plots in this appendix are designed to show the effect of pilot groups 
and the variability of results across all pilots. The ranges of the SART, TLX, DRR, and TA variables are 
provided in Table D-l. 

Table D- 1: Ranges of SART, TLX, DRR, and TA Parameters. 


# 

Variable 

Max 

Min 

1 

SART 

200 (great) 

- 1 00 (none) 

2 

TLX 

100 (high) 

0 (low) 

3 

DRR 

10 (very bad) 

1 (excellent) 

4 

TA 

100 (high) 

0 (none) 
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Figure D- 1: SART as a function of display concept and pilot type for each evaluation, VFR 

Approach. 
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Figure D- 2: SART as a function of display concept and pilot type for each evaluation, ILS 

Approach. 
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Figure D- 3: SART as a function of display concept and pilot type, VMC-like Approach. 


144 



145 


Display Concept 



TLX 


VFR Pilots 


IFR Pilots 
H-IFR Pilots 


Figure D- 5: TLX as a function of display concept and pilot type, ILS Approach. 
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Figure D- 6: TLX as a function of display concept and pilot type, VMC-like Approach. 
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Figure D- 7: DRR as a function of display concept and pilot type, VFR Approach. 
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Figure D- 8: DRR as a function of display concept and pilot type, ILS Approach. 
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Figure D- 9: DRR as a function of display concept and pilot type, VMC-like Approach. 
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Figure D- 10: Terrain awareness as a function of display concept and pilot type, VFR Approach. 
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Display Concept 



Figure D- 11: Terrain awareness as a function of display concept and pilot type, ILS Approach. 
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Figure D- 12: Terrain awareness as a function of display concept and pilot type, VMC-like 

Approach. 
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